thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-11-25 17:12:03 +00:00
Code Issues Packages Projects Releases Wiki Activity
085e8c7825
ClickHouse/benchmark/hive/log/log_100m_tuned
Ivan 97f2a2213e
Move all folders inside /dbms one level up (#9974) 
* Move some code outside dbms/src folder
* Fix paths
2020-04-02 02:51:21 +03:00

11575 lines
811 KiB
Plaintext
Raw Blame History

This file contains invisible Unicode characters

This file contains invisible Unicode characters that are indistinguishable to humans but may be processed differently by a computer. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

Not found: /etc/init.d/service with path=
start time: Вт. сент. 17 22:55:26 MSK 2013
times: 1
query: SELECT count(*) FROM hits_100m;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_18854@mturlrep13_201309172255_1961113475.txt
hive> SELECT count(*) FROM hits_100m;;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0004
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 1
2013-09-17 22:56:16,852 Stage-1 map = 0%, reduce = 0%
2013-09-17 22:56:37,989 Stage-1 map = 1%, reduce = 0%
2013-09-17 22:56:41,077 Stage-1 map = 6%, reduce = 0%
2013-09-17 22:56:43,088 Stage-1 map = 10%, reduce = 0%
2013-09-17 22:56:44,095 Stage-1 map = 14%, reduce = 0%
2013-09-17 22:56:47,115 Stage-1 map = 16%, reduce = 0%
2013-09-17 22:56:48,121 Stage-1 map = 17%, reduce = 0%
2013-09-17 22:56:50,131 Stage-1 map = 24%, reduce = 0%
2013-09-17 22:56:51,144 Stage-1 map = 25%, reduce = 0%
2013-09-17 22:56:53,160 Stage-1 map = 30%, reduce = 0%
2013-09-17 22:56:54,167 Stage-1 map = 32%, reduce = 0%
2013-09-17 22:56:55,173 Stage-1 map = 33%, reduce = 0%
2013-09-17 22:56:56,179 Stage-1 map = 36%, reduce = 0%
2013-09-17 22:56:57,185 Stage-1 map = 37%, reduce = 0%
2013-09-17 22:56:58,199 Stage-1 map = 39%, reduce = 0%
2013-09-17 22:57:00,209 Stage-1 map = 45%, reduce = 0%
2013-09-17 22:57:01,214 Stage-1 map = 46%, reduce = 0%
2013-09-17 22:57:03,223 Stage-1 map = 51%, reduce = 0%
2013-09-17 22:57:04,230 Stage-1 map = 52%, reduce = 0%
2013-09-17 22:57:05,693 Stage-1 map = 53%, reduce = 0%, Cumulative CPU 36.54 sec
2013-09-17 22:57:06,700 Stage-1 map = 57%, reduce = 0%, Cumulative CPU 36.54 sec
2013-09-17 22:57:07,707 Stage-1 map = 58%, reduce = 0%, Cumulative CPU 36.54 sec
2013-09-17 22:57:08,725 Stage-1 map = 58%, reduce = 0%, Cumulative CPU 36.54 sec
2013-09-17 22:57:09,731 Stage-1 map = 59%, reduce = 0%, Cumulative CPU 36.54 sec
2013-09-17 22:57:10,948 Stage-1 map = 63%, reduce = 0%, Cumulative CPU 36.54 sec
2013-09-17 22:57:11,958 Stage-1 map = 65%, reduce = 0%, Cumulative CPU 36.54 sec
2013-09-17 22:57:12,970 Stage-1 map = 69%, reduce = 0%, Cumulative CPU 36.54 sec
2013-09-17 22:57:13,990 Stage-1 map = 73%, reduce = 0%, Cumulative CPU 1483.14 sec
2013-09-17 22:57:15,025 Stage-1 map = 74%, reduce = 0%, Cumulative CPU 1484.59 sec
2013-09-17 22:57:16,048 Stage-1 map = 75%, reduce = 0%, Cumulative CPU 1484.59 sec
2013-09-17 22:57:17,060 Stage-1 map = 76%, reduce = 0%, Cumulative CPU 1484.59 sec
2013-09-17 22:57:18,095 Stage-1 map = 79%, reduce = 0%, Cumulative CPU 1502.09 sec
2013-09-17 22:57:19,123 Stage-1 map = 84%, reduce = 0%, Cumulative CPU 1512.83 sec
2013-09-17 22:57:20,133 Stage-1 map = 87%, reduce = 0%, Cumulative CPU 1530.9 sec
2013-09-17 22:57:21,147 Stage-1 map = 90%, reduce = 0%, Cumulative CPU 1580.07 sec
2013-09-17 22:57:22,153 Stage-1 map = 94%, reduce = 0%, Cumulative CPU 1640.62 sec
2013-09-17 22:57:23,159 Stage-1 map = 95%, reduce = 0%, Cumulative CPU 1677.56 sec
2013-09-17 22:57:24,166 Stage-1 map = 97%, reduce = 0%, Cumulative CPU 1722.04 sec
2013-09-17 22:57:25,267 Stage-1 map = 98%, reduce = 0%, Cumulative CPU 1733.88 sec
2013-09-17 22:57:26,274 Stage-1 map = 99%, reduce = 0%, Cumulative CPU 1760.11 sec
2013-09-17 22:57:27,280 Stage-1 map = 99%, reduce = 0%, Cumulative CPU 1772.82 sec
2013-09-17 22:57:28,286 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1786.49 sec
2013-09-17 22:57:29,291 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1786.49 sec
2013-09-17 22:57:30,297 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1786.49 sec
2013-09-17 22:57:31,302 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1786.49 sec
2013-09-17 22:57:32,307 Stage-1 map = 100%, reduce = 32%, Cumulative CPU 1786.49 sec
2013-09-17 22:57:33,313 Stage-1 map = 100%, reduce = 32%, Cumulative CPU 1786.49 sec
2013-09-17 22:57:34,319 Stage-1 map = 100%, reduce = 32%, Cumulative CPU 1786.49 sec
2013-09-17 22:57:35,337 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 1789.45 sec
2013-09-17 22:57:36,344 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 1789.45 sec
2013-09-17 22:57:37,350 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 1789.45 sec
MapReduce Total cumulative CPU time: 29 minutes 49 seconds 450 msec
Ended Job = job_201309172235_0004
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 1 Cumulative CPU: 1789.45 sec HDFS Read: 10081441390 HDFS Write: 10 SUCCESS
Total MapReduce CPU Time Spent: 29 minutes 49 seconds 450 msec
OK
100000000
Time taken: 110.676 seconds, Fetched: 1 row(s)
hive> quit;
times: 2
query: SELECT count(*) FROM hits_100m;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_23942@mturlrep13_201309172257_1656900452.txt
hive> SELECT count(*) FROM hits_100m;;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0005
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 1
2013-09-17 22:58:08,971 Stage-1 map = 0%, reduce = 0%
2013-09-17 22:58:29,310 Stage-1 map = 1%, reduce = 0%
2013-09-17 22:58:32,327 Stage-1 map = 4%, reduce = 0%
2013-09-17 22:58:33,338 Stage-1 map = 5%, reduce = 0%
2013-09-17 22:58:34,345 Stage-1 map = 6%, reduce = 0%
2013-09-17 22:58:35,354 Stage-1 map = 8%, reduce = 0%
2013-09-17 22:58:36,360 Stage-1 map = 13%, reduce = 0%
2013-09-17 22:58:37,376 Stage-1 map = 15%, reduce = 0%
2013-09-17 22:58:39,389 Stage-1 map = 17%, reduce = 0%
2013-09-17 22:58:40,408 Stage-1 map = 18%, reduce = 0%
2013-09-17 22:58:41,420 Stage-1 map = 20%, reduce = 0%
2013-09-17 22:58:42,432 Stage-1 map = 25%, reduce = 0%
2013-09-17 22:58:43,444 Stage-1 map = 27%, reduce = 0%
2013-09-17 22:58:44,461 Stage-1 map = 29%, reduce = 0%
2013-09-17 22:58:45,468 Stage-1 map = 31%, reduce = 0%
2013-09-17 22:58:46,488 Stage-1 map = 33%, reduce = 0%
2013-09-17 22:58:48,499 Stage-1 map = 36%, reduce = 0%
2013-09-17 22:58:49,507 Stage-1 map = 38%, reduce = 0%
2013-09-17 22:58:50,513 Stage-1 map = 41%, reduce = 0%
2013-09-17 22:58:51,519 Stage-1 map = 42%, reduce = 0%
2013-09-17 22:58:52,524 Stage-1 map = 45%, reduce = 0%
2013-09-17 22:58:53,529 Stage-1 map = 46%, reduce = 0%
2013-09-17 22:58:54,535 Stage-1 map = 48%, reduce = 0%
2013-09-17 22:58:55,575 Stage-1 map = 51%, reduce = 0%
2013-09-17 22:58:56,602 Stage-1 map = 53%, reduce = 0%
2013-09-17 22:58:57,639 Stage-1 map = 54%, reduce = 0%
2013-09-17 22:58:58,645 Stage-1 map = 57%, reduce = 0%
2013-09-17 22:58:59,664 Stage-1 map = 59%, reduce = 0%
2013-09-17 22:59:01,690 Stage-1 map = 65%, reduce = 0%
2013-09-17 22:59:02,722 Stage-1 map = 68%, reduce = 0%, Cumulative CPU 36.1 sec
2013-09-17 22:59:03,750 Stage-1 map = 69%, reduce = 0%, Cumulative CPU 36.1 sec
2013-09-17 22:59:04,761 Stage-1 map = 71%, reduce = 0%, Cumulative CPU 36.1 sec
2013-09-17 22:59:05,770 Stage-1 map = 73%, reduce = 0%, Cumulative CPU 36.1 sec
2013-09-17 22:59:06,784 Stage-1 map = 73%, reduce = 0%, Cumulative CPU 36.1 sec
2013-09-17 22:59:08,117 Stage-1 map = 80%, reduce = 0%, Cumulative CPU 175.85 sec
2013-09-17 22:59:09,124 Stage-1 map = 81%, reduce = 0%, Cumulative CPU 221.51 sec
2013-09-17 22:59:10,130 Stage-1 map = 83%, reduce = 0%, Cumulative CPU 314.14 sec
2013-09-17 22:59:11,138 Stage-1 map = 88%, reduce = 0%, Cumulative CPU 650.01 sec
2013-09-17 22:59:12,145 Stage-1 map = 90%, reduce = 0%, Cumulative CPU 788.39 sec
2013-09-17 22:59:13,149 Stage-1 map = 94%, reduce = 0%, Cumulative CPU 1123.31 sec
2013-09-17 22:59:14,155 Stage-1 map = 96%, reduce = 0%, Cumulative CPU 1744.89 sec
2013-09-17 22:59:15,161 Stage-1 map = 98%, reduce = 0%, Cumulative CPU 1753.27 sec
2013-09-17 22:59:16,167 Stage-1 map = 99%, reduce = 0%, Cumulative CPU 1760.31 sec
2013-09-17 22:59:17,174 Stage-1 map = 99%, reduce = 0%, Cumulative CPU 1765.77 sec
2013-09-17 22:59:18,179 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1773.76 sec
2013-09-17 22:59:19,185 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1773.76 sec
2013-09-17 22:59:20,191 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1773.76 sec
2013-09-17 22:59:21,196 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1773.76 sec
2013-09-17 22:59:22,202 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1773.76 sec
2013-09-17 22:59:23,215 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1773.76 sec
2013-09-17 22:59:24,221 Stage-1 map = 100%, reduce = 32%, Cumulative CPU 1773.76 sec
2013-09-17 22:59:25,226 Stage-1 map = 100%, reduce = 32%, Cumulative CPU 1773.76 sec
2013-09-17 22:59:26,234 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 1776.78 sec
2013-09-17 22:59:27,552 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 1776.78 sec
MapReduce Total cumulative CPU time: 29 minutes 36 seconds 780 msec
Ended Job = job_201309172235_0005
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 1 Cumulative CPU: 1776.78 sec HDFS Read: 10081441390 HDFS Write: 10 SUCCESS
Total MapReduce CPU Time Spent: 29 minutes 36 seconds 780 msec
OK
100000000
Time taken: 105.13 seconds, Fetched: 1 row(s)
hive> quit;
times: 3
query: SELECT count(*) FROM hits_100m;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_28935@mturlrep13_201309172259_1582946547.txt
hive> SELECT count(*) FROM hits_100m;;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0006
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 1
2013-09-17 22:59:59,626 Stage-1 map = 0%, reduce = 0%
2013-09-17 23:00:21,889 Stage-1 map = 1%, reduce = 0%, Cumulative CPU 106.84 sec
2013-09-17 23:00:22,898 Stage-1 map = 1%, reduce = 0%, Cumulative CPU 106.84 sec
2013-09-17 23:00:24,158 Stage-1 map = 5%, reduce = 0%, Cumulative CPU 106.84 sec
2013-09-17 23:00:25,172 Stage-1 map = 5%, reduce = 0%, Cumulative CPU 106.84 sec
2013-09-17 23:00:26,201 Stage-1 map = 6%, reduce = 0%, Cumulative CPU 106.84 sec
2013-09-17 23:00:27,208 Stage-1 map = 12%, reduce = 0%, Cumulative CPU 106.84 sec
2013-09-17 23:00:28,215 Stage-1 map = 13%, reduce = 0%, Cumulative CPU 106.84 sec
2013-09-17 23:00:29,226 Stage-1 map = 16%, reduce = 0%, Cumulative CPU 106.84 sec
2013-09-17 23:00:30,236 Stage-1 map = 18%, reduce = 0%, Cumulative CPU 106.84 sec
2013-09-17 23:00:31,983 Stage-1 map = 18%, reduce = 0%, Cumulative CPU 106.84 sec
2013-09-17 23:00:32,998 Stage-1 map = 22%, reduce = 0%, Cumulative CPU 106.84 sec
2013-09-17 23:00:34,970 Stage-1 map = 24%, reduce = 0%, Cumulative CPU 106.84 sec
2013-09-17 23:00:35,996 Stage-1 map = 28%, reduce = 0%, Cumulative CPU 106.84 sec
2013-09-17 23:00:37,017 Stage-1 map = 29%, reduce = 0%, Cumulative CPU 106.84 sec
2013-09-17 23:00:38,034 Stage-1 map = 30%, reduce = 0%, Cumulative CPU 106.84 sec
2013-09-17 23:00:39,050 Stage-1 map = 31%, reduce = 0%, Cumulative CPU 106.84 sec
2013-09-17 23:00:40,078 Stage-1 map = 34%, reduce = 0%, Cumulative CPU 106.84 sec
2013-09-17 23:00:41,092 Stage-1 map = 36%, reduce = 0%, Cumulative CPU 106.84 sec
2013-09-17 23:00:42,097 Stage-1 map = 39%, reduce = 0%, Cumulative CPU 106.84 sec
2013-09-17 23:00:43,104 Stage-1 map = 41%, reduce = 0%, Cumulative CPU 106.84 sec
2013-09-17 23:00:44,109 Stage-1 map = 43%, reduce = 0%, Cumulative CPU 106.84 sec
2013-09-17 23:00:45,115 Stage-1 map = 46%, reduce = 0%, Cumulative CPU 106.84 sec
2013-09-17 23:00:46,124 Stage-1 map = 48%, reduce = 0%, Cumulative CPU 106.84 sec
2013-09-17 23:00:47,148 Stage-1 map = 50%, reduce = 0%, Cumulative CPU 106.84 sec
2013-09-17 23:00:48,154 Stage-1 map = 50%, reduce = 0%, Cumulative CPU 106.84 sec
2013-09-17 23:00:49,162 Stage-1 map = 53%, reduce = 0%, Cumulative CPU 106.84 sec
2013-09-17 23:00:50,172 Stage-1 map = 56%, reduce = 0%, Cumulative CPU 106.84 sec
2013-09-17 23:00:51,207 Stage-1 map = 57%, reduce = 0%, Cumulative CPU 140.29 sec
2013-09-17 23:00:52,224 Stage-1 map = 62%, reduce = 0%, Cumulative CPU 140.29 sec
2013-09-17 23:00:53,230 Stage-1 map = 64%, reduce = 0%, Cumulative CPU 140.29 sec
2013-09-17 23:00:54,237 Stage-1 map = 65%, reduce = 0%, Cumulative CPU 140.29 sec
2013-09-17 23:00:55,247 Stage-1 map = 67%, reduce = 0%, Cumulative CPU 140.29 sec
2013-09-17 23:00:56,254 Stage-1 map = 68%, reduce = 0%, Cumulative CPU 140.29 sec
2013-09-17 23:00:57,261 Stage-1 map = 69%, reduce = 0%, Cumulative CPU 140.29 sec
2013-09-17 23:00:58,268 Stage-1 map = 74%, reduce = 0%, Cumulative CPU 178.19 sec
2013-09-17 23:00:59,275 Stage-1 map = 76%, reduce = 0%, Cumulative CPU 178.19 sec
2013-09-17 23:01:00,282 Stage-1 map = 77%, reduce = 0%, Cumulative CPU 178.19 sec
2013-09-17 23:01:01,288 Stage-1 map = 79%, reduce = 0%, Cumulative CPU 309.14 sec
2013-09-17 23:01:02,448 Stage-1 map = 84%, reduce = 0%, Cumulative CPU 570.2 sec
2013-09-17 23:01:03,547 Stage-1 map = 86%, reduce = 0%, Cumulative CPU 708.2 sec
2013-09-17 23:01:04,555 Stage-1 map = 90%, reduce = 0%, Cumulative CPU 845.2 sec
2013-09-17 23:01:05,562 Stage-1 map = 92%, reduce = 0%, Cumulative CPU 982.42 sec
2013-09-17 23:01:06,568 Stage-1 map = 95%, reduce = 0%, Cumulative CPU 1312.62 sec
2013-09-17 23:01:07,574 Stage-1 map = 98%, reduce = 0%, Cumulative CPU 1599.05 sec
2013-09-17 23:01:08,580 Stage-1 map = 98%, reduce = 0%, Cumulative CPU 1599.05 sec
2013-09-17 23:01:09,585 Stage-1 map = 99%, reduce = 0%, Cumulative CPU 1697.63 sec
2013-09-17 23:01:10,590 Stage-1 map = 99%, reduce = 0%, Cumulative CPU 1749.84 sec
2013-09-17 23:01:11,595 Stage-1 map = 99%, reduce = 0%, Cumulative CPU 1749.84 sec
2013-09-17 23:01:12,601 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1802.81 sec
2013-09-17 23:01:13,606 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1802.81 sec
2013-09-17 23:01:14,611 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1803.33 sec
2013-09-17 23:01:15,617 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 1803.33 sec
2013-09-17 23:01:16,624 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 1803.33 sec
2013-09-17 23:01:17,632 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 1805.73 sec
2013-09-17 23:01:18,638 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 1805.73 sec
2013-09-17 23:01:19,644 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 1805.73 sec
MapReduce Total cumulative CPU time: 30 minutes 5 seconds 730 msec
Ended Job = job_201309172235_0006
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 1 Cumulative CPU: 1805.73 sec HDFS Read: 10081441390 HDFS Write: 10 SUCCESS
Total MapReduce CPU Time Spent: 30 minutes 5 seconds 730 msec
OK
100000000
Time taken: 107.358 seconds, Fetched: 1 row(s)
hive> quit;
times: 1
query: SELECT count(*) FROM hits_100m WHERE AdvEngineID != 0;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_2345@mturlrep13_201309172301_1950916943.txt
hive> SELECT count(*) FROM hits_100m WHERE AdvEngineID != 0;;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0007
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 1
2013-09-17 23:02:11,138 Stage-1 map = 0%, reduce = 0%
2013-09-17 23:02:22,387 Stage-1 map = 4%, reduce = 0%
2013-09-17 23:02:23,396 Stage-1 map = 11%, reduce = 0%
2013-09-17 23:02:24,402 Stage-1 map = 30%, reduce = 0%
2013-09-17 23:02:25,700 Stage-1 map = 35%, reduce = 0%
2013-09-17 23:02:26,723 Stage-1 map = 57%, reduce = 0%, Cumulative CPU 46.24 sec
2013-09-17 23:02:27,736 Stage-1 map = 82%, reduce = 0%, Cumulative CPU 173.19 sec
2013-09-17 23:02:28,746 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 294.36 sec
2013-09-17 23:02:29,751 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 294.36 sec
2013-09-17 23:02:30,756 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 294.36 sec
2013-09-17 23:02:31,761 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 294.36 sec
2013-09-17 23:02:32,768 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 294.36 sec
2013-09-17 23:02:33,774 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 294.36 sec
2013-09-17 23:02:34,943 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 294.36 sec
2013-09-17 23:02:35,950 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 294.36 sec
2013-09-17 23:02:36,959 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 297.32 sec
2013-09-17 23:02:37,967 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 297.32 sec
MapReduce Total cumulative CPU time: 4 minutes 57 seconds 320 msec
Ended Job = job_201309172235_0007
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 1 Cumulative CPU: 297.32 sec HDFS Read: 6282387 HDFS Write: 7 SUCCESS
Total MapReduce CPU Time Spent: 4 minutes 57 seconds 320 msec
OK
630535
Time taken: 55.195 seconds, Fetched: 1 row(s)
hive> quit;
times: 2
query: SELECT count(*) FROM hits_100m WHERE AdvEngineID != 0;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_7709@mturlrep13_201309172302_964147901.txt
hive> SELECT count(*) FROM hits_100m WHERE AdvEngineID != 0;;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0008
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 1
2013-09-17 23:02:52,920 Stage-1 map = 0%, reduce = 0%
2013-09-17 23:03:03,995 Stage-1 map = 2%, reduce = 0%
2013-09-17 23:03:05,016 Stage-1 map = 3%, reduce = 0%
2013-09-17 23:03:06,076 Stage-1 map = 48%, reduce = 0%, Cumulative CPU 6.7 sec
2013-09-17 23:03:07,088 Stage-1 map = 88%, reduce = 0%, Cumulative CPU 210.4 sec
2013-09-17 23:03:08,096 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 294.76 sec
2013-09-17 23:03:09,102 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 294.76 sec
2013-09-17 23:03:10,109 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 294.76 sec
2013-09-17 23:03:11,114 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 294.76 sec
2013-09-17 23:03:12,127 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 294.76 sec
2013-09-17 23:03:13,133 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 294.76 sec
2013-09-17 23:03:14,139 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 294.76 sec
2013-09-17 23:03:15,146 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 295.45 sec
2013-09-17 23:03:16,154 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 297.51 sec
2013-09-17 23:03:17,160 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 297.51 sec
MapReduce Total cumulative CPU time: 4 minutes 57 seconds 510 msec
Ended Job = job_201309172235_0008
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 1 Cumulative CPU: 297.51 sec HDFS Read: 6282387 HDFS Write: 7 SUCCESS
Total MapReduce CPU Time Spent: 4 minutes 57 seconds 510 msec
OK
630535
Time taken: 36.435 seconds, Fetched: 1 row(s)
hive> quit;
times: 3
query: SELECT count(*) FROM hits_100m WHERE AdvEngineID != 0;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_12665@mturlrep13_201309172303_2013045725.txt
hive> SELECT count(*) FROM hits_100m WHERE AdvEngineID != 0;;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0009
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 1
2013-09-17 23:03:27,027 Stage-1 map = 0%, reduce = 0%
2013-09-17 23:03:37,093 Stage-1 map = 1%, reduce = 0%
2013-09-17 23:03:40,143 Stage-1 map = 41%, reduce = 0%
2013-09-17 23:03:41,161 Stage-1 map = 83%, reduce = 0%, Cumulative CPU 171.9 sec
2013-09-17 23:03:42,169 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 298.73 sec
2013-09-17 23:03:43,177 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 298.73 sec
2013-09-17 23:03:44,184 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 298.73 sec
2013-09-17 23:03:45,190 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 298.73 sec
2013-09-17 23:03:46,196 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 298.73 sec
2013-09-17 23:03:47,202 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 298.73 sec
2013-09-17 23:03:48,208 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 298.73 sec
2013-09-17 23:03:49,214 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 298.73 sec
2013-09-17 23:03:50,223 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 301.9 sec
2013-09-17 23:03:51,229 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 301.9 sec
MapReduce Total cumulative CPU time: 5 minutes 1 seconds 900 msec
Ended Job = job_201309172235_0009
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 1 Cumulative CPU: 301.9 sec HDFS Read: 6282387 HDFS Write: 7 SUCCESS
Total MapReduce CPU Time Spent: 5 minutes 1 seconds 900 msec
OK
630535
Time taken: 32.201 seconds, Fetched: 1 row(s)
hive> quit;
times: 1
query: SELECT sum(AdvEngineID), count(*), avg(ResolutionWidth) FROM hits_100m;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_17646@mturlrep13_201309172303_809195796.txt
hive> SELECT sum(AdvEngineID), count(*), avg(ResolutionWidth) FROM hits_100m;;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0010
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 1
2013-09-17 23:04:11,044 Stage-1 map = 0%, reduce = 0%
2013-09-17 23:04:30,217 Stage-1 map = 21%, reduce = 0%, Cumulative CPU 29.16 sec
2013-09-17 23:04:31,225 Stage-1 map = 35%, reduce = 0%, Cumulative CPU 90.03 sec
2013-09-17 23:04:32,231 Stage-1 map = 64%, reduce = 0%, Cumulative CPU 230.36 sec
2013-09-17 23:04:33,237 Stage-1 map = 97%, reduce = 0%, Cumulative CPU 349.12 sec
2013-09-17 23:04:34,243 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 391.27 sec
2013-09-17 23:04:35,249 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 391.27 sec
2013-09-17 23:04:36,255 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 391.27 sec
2013-09-17 23:04:37,261 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 391.27 sec
2013-09-17 23:04:38,267 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 391.27 sec
2013-09-17 23:04:39,276 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 393.94 sec
2013-09-17 23:04:40,283 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 393.94 sec
MapReduce Total cumulative CPU time: 6 minutes 33 seconds 940 msec
Ended Job = job_201309172235_0010
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 1 Cumulative CPU: 393.94 sec HDFS Read: 78754487 HDFS Write: 32 SUCCESS
Total MapReduce CPU Time Spent: 6 minutes 33 seconds 940 msec
OK
Time taken: 39.991 seconds, Fetched: 1 row(s)
hive> quit;
times: 2
query: SELECT sum(AdvEngineID), count(*), avg(ResolutionWidth) FROM hits_100m;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_22907@mturlrep13_201309172304_403247887.txt
hive> SELECT sum(AdvEngineID), count(*), avg(ResolutionWidth) FROM hits_100m;;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0011
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 1
2013-09-17 23:04:50,128 Stage-1 map = 0%, reduce = 0%
2013-09-17 23:05:02,260 Stage-1 map = 2%, reduce = 0%
2013-09-17 23:05:03,278 Stage-1 map = 13%, reduce = 0%
2013-09-17 23:05:04,299 Stage-1 map = 25%, reduce = 0%
2013-09-17 23:05:05,308 Stage-1 map = 30%, reduce = 0%
2013-09-17 23:05:06,333 Stage-1 map = 52%, reduce = 0%, Cumulative CPU 9.21 sec
2013-09-17 23:05:07,352 Stage-1 map = 70%, reduce = 0%, Cumulative CPU 39.3 sec
2013-09-17 23:05:08,369 Stage-1 map = 98%, reduce = 0%, Cumulative CPU 377.82 sec
2013-09-17 23:05:09,375 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 401.7 sec
2013-09-17 23:05:10,381 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 401.7 sec
2013-09-17 23:05:11,387 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 401.7 sec
2013-09-17 23:05:12,393 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 401.7 sec
2013-09-17 23:05:13,399 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 401.7 sec
2013-09-17 23:05:14,405 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 401.7 sec
2013-09-17 23:05:15,411 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 401.7 sec
2013-09-17 23:05:16,419 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 404.35 sec
2013-09-17 23:05:17,426 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 404.35 sec
MapReduce Total cumulative CPU time: 6 minutes 44 seconds 350 msec
Ended Job = job_201309172235_0011
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 1 Cumulative CPU: 404.35 sec HDFS Read: 78754487 HDFS Write: 32 SUCCESS
Total MapReduce CPU Time Spent: 6 minutes 44 seconds 350 msec
OK
Time taken: 35.143 seconds, Fetched: 1 row(s)
hive> quit;
times: 3
query: SELECT sum(AdvEngineID), count(*), avg(ResolutionWidth) FROM hits_100m;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_28807@mturlrep13_201309172305_1471136727.txt
hive> SELECT sum(AdvEngineID), count(*), avg(ResolutionWidth) FROM hits_100m;;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0012
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 1
2013-09-17 23:05:28,152 Stage-1 map = 0%, reduce = 0%
2013-09-17 23:05:40,255 Stage-1 map = 7%, reduce = 0%
2013-09-17 23:05:41,260 Stage-1 map = 22%, reduce = 0%
2013-09-17 23:05:42,266 Stage-1 map = 28%, reduce = 0%
2013-09-17 23:05:43,279 Stage-1 map = 39%, reduce = 0%, Cumulative CPU 9.21 sec
2013-09-17 23:05:44,286 Stage-1 map = 72%, reduce = 0%, Cumulative CPU 90.45 sec
2013-09-17 23:05:45,294 Stage-1 map = 97%, reduce = 0%, Cumulative CPU 363.6 sec
2013-09-17 23:05:46,301 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 399.15 sec
2013-09-17 23:05:47,307 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 399.15 sec
2013-09-17 23:05:48,312 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 399.15 sec
2013-09-17 23:05:49,317 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 399.15 sec
2013-09-17 23:05:50,323 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 399.15 sec
2013-09-17 23:05:51,329 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 399.15 sec
2013-09-17 23:05:52,334 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 399.15 sec
2013-09-17 23:05:53,343 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 402.09 sec
2013-09-17 23:05:54,348 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 402.09 sec
MapReduce Total cumulative CPU time: 6 minutes 42 seconds 90 msec
Ended Job = job_201309172235_0012
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 1 Cumulative CPU: 402.09 sec HDFS Read: 78754487 HDFS Write: 32 SUCCESS
Total MapReduce CPU Time Spent: 6 minutes 42 seconds 90 msec
OK
Time taken: 35.085 seconds, Fetched: 1 row(s)
hive> quit;
times: 1
query: SELECT sum(UserID) FROM hits_100m;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_1699@mturlrep13_201309172306_1224643663.txt
hive> SELECT sum(UserID) FROM hits_100m;;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0013
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 1
2013-09-17 23:06:12,215 Stage-1 map = 0%, reduce = 0%
2013-09-17 23:06:22,307 Stage-1 map = 2%, reduce = 0%
2013-09-17 23:06:24,332 Stage-1 map = 5%, reduce = 0%
2013-09-17 23:06:26,810 Stage-1 map = 15%, reduce = 0%
2013-09-17 23:06:27,822 Stage-1 map = 17%, reduce = 0%, Cumulative CPU 7.28 sec
2013-09-17 23:06:28,829 Stage-1 map = 17%, reduce = 0%, Cumulative CPU 7.28 sec
2013-09-17 23:06:29,837 Stage-1 map = 17%, reduce = 0%, Cumulative CPU 7.28 sec
2013-09-17 23:06:32,109 Stage-1 map = 44%, reduce = 0%, Cumulative CPU 25.3 sec
2013-09-17 23:06:33,115 Stage-1 map = 44%, reduce = 0%, Cumulative CPU 25.3 sec
2013-09-17 23:06:35,953 Stage-1 map = 70%, reduce = 0%, Cumulative CPU 34.74 sec
2013-09-17 23:06:36,962 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 352.67 sec
2013-09-17 23:06:37,968 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 352.67 sec
2013-09-17 23:06:38,974 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 352.67 sec
2013-09-17 23:06:39,979 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 352.67 sec
2013-09-17 23:06:40,985 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 352.67 sec
2013-09-17 23:06:41,991 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 352.67 sec
2013-09-17 23:06:42,997 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 352.67 sec
2013-09-17 23:06:44,006 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 355.35 sec
2013-09-17 23:06:45,013 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 355.35 sec
2013-09-17 23:06:46,025 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 355.35 sec
MapReduce Total cumulative CPU time: 5 minutes 55 seconds 350 msec
Ended Job = job_201309172235_0013
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 1 Cumulative CPU: 355.35 sec HDFS Read: 663537300 HDFS Write: 21 SUCCESS
Total MapReduce CPU Time Spent: 5 minutes 55 seconds 350 msec
OK
-2020069000640784227
Time taken: 44.465 seconds, Fetched: 1 row(s)
hive> quit;
times: 2
query: SELECT sum(UserID) FROM hits_100m;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_7091@mturlrep13_201309172306_2064459587.txt
hive> SELECT sum(UserID) FROM hits_100m;;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0014
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 1
2013-09-17 23:06:55,801 Stage-1 map = 0%, reduce = 0%
2013-09-17 23:07:07,899 Stage-1 map = 1%, reduce = 0%
2013-09-17 23:07:08,910 Stage-1 map = 20%, reduce = 0%
2013-09-17 23:07:09,919 Stage-1 map = 38%, reduce = 0%
2013-09-17 23:07:10,933 Stage-1 map = 45%, reduce = 0%, Cumulative CPU 26.96 sec
2013-09-17 23:07:11,941 Stage-1 map = 90%, reduce = 0%, Cumulative CPU 277.91 sec
2013-09-17 23:07:12,948 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 349.42 sec
2013-09-17 23:07:13,954 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 349.42 sec
2013-09-17 23:07:14,960 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 349.42 sec
2013-09-17 23:07:15,966 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 349.42 sec
2013-09-17 23:07:16,972 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 349.42 sec
2013-09-17 23:07:17,978 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 349.42 sec
2013-09-17 23:07:18,984 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 349.42 sec
2013-09-17 23:07:19,992 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 352.35 sec
2013-09-17 23:07:20,998 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 352.35 sec
2013-09-17 23:07:22,005 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 352.35 sec
MapReduce Total cumulative CPU time: 5 minutes 52 seconds 350 msec
Ended Job = job_201309172235_0014
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 1 Cumulative CPU: 352.35 sec HDFS Read: 663537300 HDFS Write: 21 SUCCESS
Total MapReduce CPU Time Spent: 5 minutes 52 seconds 350 msec
OK
-2020069000640784227
Time taken: 34.131 seconds, Fetched: 1 row(s)
hive> quit;
times: 3
query: SELECT sum(UserID) FROM hits_100m;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_12067@mturlrep13_201309172307_240829598.txt
hive> SELECT sum(UserID) FROM hits_100m;;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0015
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 1
2013-09-17 23:07:31,722 Stage-1 map = 0%, reduce = 0%
2013-09-17 23:07:43,798 Stage-1 map = 3%, reduce = 0%
2013-09-17 23:07:44,804 Stage-1 map = 24%, reduce = 0%
2013-09-17 23:07:45,810 Stage-1 map = 38%, reduce = 0%
2013-09-17 23:07:46,822 Stage-1 map = 46%, reduce = 0%, Cumulative CPU 33.64 sec
2013-09-17 23:07:47,830 Stage-1 map = 86%, reduce = 0%, Cumulative CPU 257.18 sec
2013-09-17 23:07:48,837 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 349.23 sec
2013-09-17 23:07:49,842 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 349.23 sec
2013-09-17 23:07:50,847 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 349.23 sec
2013-09-17 23:07:51,852 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 349.23 sec
2013-09-17 23:07:52,858 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 349.23 sec
2013-09-17 23:07:53,863 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 349.23 sec
2013-09-17 23:07:54,869 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 349.23 sec
2013-09-17 23:07:55,877 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 352.02 sec
2013-09-17 23:07:56,884 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 352.02 sec
2013-09-17 23:07:57,889 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 352.02 sec
MapReduce Total cumulative CPU time: 5 minutes 52 seconds 20 msec
Ended Job = job_201309172235_0015
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 1 Cumulative CPU: 352.02 sec HDFS Read: 663537300 HDFS Write: 21 SUCCESS
Total MapReduce CPU Time Spent: 5 minutes 52 seconds 20 msec
OK
-2020069000640784227
Time taken: 34.032 seconds, Fetched: 1 row(s)
hive> quit;
times: 1
query: SELECT count(DISTINCT UserID) FROM hits_100m;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_17039@mturlrep13_201309172308_800014414.txt
hive> SELECT count(DISTINCT UserID) FROM hits_100m;;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0016
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 1
2013-09-17 23:08:15,876 Stage-1 map = 0%, reduce = 0%
2013-09-17 23:08:25,047 Stage-1 map = 1%, reduce = 0%
2013-09-17 23:08:27,066 Stage-1 map = 2%, reduce = 0%
2013-09-17 23:08:28,077 Stage-1 map = 6%, reduce = 0%
2013-09-17 23:08:29,083 Stage-1 map = 11%, reduce = 0%
2013-09-17 23:08:30,088 Stage-1 map = 18%, reduce = 0%
2013-09-17 23:08:31,110 Stage-1 map = 19%, reduce = 0%
2013-09-17 23:08:32,125 Stage-1 map = 29%, reduce = 0%
2013-09-17 23:08:33,130 Stage-1 map = 34%, reduce = 0%
2013-09-17 23:08:35,158 Stage-1 map = 47%, reduce = 0%
2013-09-17 23:08:36,166 Stage-1 map = 51%, reduce = 0%
2013-09-17 23:08:38,177 Stage-1 map = 65%, reduce = 0%
2013-09-17 23:08:39,189 Stage-1 map = 71%, reduce = 0%, Cumulative CPU 29.98 sec
2013-09-17 23:08:40,216 Stage-1 map = 73%, reduce = 0%, Cumulative CPU 101.62 sec
2013-09-17 23:08:41,230 Stage-1 map = 79%, reduce = 0%, Cumulative CPU 101.62 sec
2013-09-17 23:08:42,264 Stage-1 map = 82%, reduce = 0%, Cumulative CPU 101.62 sec
2013-09-17 23:08:43,274 Stage-1 map = 84%, reduce = 0%, Cumulative CPU 179.43 sec
2013-09-17 23:08:44,289 Stage-1 map = 87%, reduce = 0%, Cumulative CPU 250.32 sec
2013-09-17 23:08:45,300 Stage-1 map = 90%, reduce = 0%, Cumulative CPU 338.89 sec
2013-09-17 23:08:46,306 Stage-1 map = 92%, reduce = 0%, Cumulative CPU 456.84 sec
2013-09-17 23:08:47,312 Stage-1 map = 95%, reduce = 0%, Cumulative CPU 496.19 sec
2013-09-17 23:08:48,318 Stage-1 map = 97%, reduce = 0%, Cumulative CPU 522.86 sec
2013-09-17 23:08:49,323 Stage-1 map = 98%, reduce = 1%, Cumulative CPU 600.26 sec
2013-09-17 23:08:50,329 Stage-1 map = 99%, reduce = 1%, Cumulative CPU 653.17 sec
2013-09-17 23:08:51,335 Stage-1 map = 99%, reduce = 1%, Cumulative CPU 653.17 sec
2013-09-17 23:08:52,341 Stage-1 map = 100%, reduce = 30%, Cumulative CPU 708.39 sec
2013-09-17 23:08:53,347 Stage-1 map = 100%, reduce = 30%, Cumulative CPU 708.39 sec
2013-09-17 23:08:54,352 Stage-1 map = 100%, reduce = 30%, Cumulative CPU 708.39 sec
2013-09-17 23:08:55,359 Stage-1 map = 100%, reduce = 30%, Cumulative CPU 708.39 sec
2013-09-17 23:08:56,365 Stage-1 map = 100%, reduce = 30%, Cumulative CPU 708.39 sec
2013-09-17 23:08:57,371 Stage-1 map = 100%, reduce = 30%, Cumulative CPU 708.39 sec
2013-09-17 23:08:58,377 Stage-1 map = 100%, reduce = 30%, Cumulative CPU 708.39 sec
2013-09-17 23:08:59,383 Stage-1 map = 100%, reduce = 30%, Cumulative CPU 708.39 sec
2013-09-17 23:09:00,389 Stage-1 map = 100%, reduce = 30%, Cumulative CPU 708.39 sec
2013-09-17 23:09:01,395 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 708.39 sec
2013-09-17 23:09:02,401 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 708.39 sec
2013-09-17 23:09:03,406 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 708.39 sec
2013-09-17 23:09:04,412 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 708.39 sec
2013-09-17 23:09:05,418 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 708.39 sec
2013-09-17 23:09:06,424 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 708.39 sec
2013-09-17 23:09:07,430 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 708.39 sec
2013-09-17 23:09:08,436 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 708.39 sec
2013-09-17 23:09:09,442 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 708.39 sec
2013-09-17 23:09:10,447 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 708.39 sec
2013-09-17 23:09:11,453 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 708.39 sec
2013-09-17 23:09:12,459 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 708.39 sec
2013-09-17 23:09:13,464 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 708.39 sec
2013-09-17 23:09:14,470 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 708.39 sec
2013-09-17 23:09:15,476 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 733.92 sec
2013-09-17 23:09:16,482 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 733.92 sec
2013-09-17 23:09:17,488 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 733.92 sec
2013-09-17 23:09:18,494 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 733.92 sec
2013-09-17 23:09:19,500 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 733.92 sec
2013-09-17 23:09:20,506 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 733.92 sec
2013-09-17 23:09:21,511 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 733.92 sec
2013-09-17 23:09:22,517 Stage-1 map = 100%, reduce = 68%, Cumulative CPU 733.92 sec
2013-09-17 23:09:23,523 Stage-1 map = 100%, reduce = 68%, Cumulative CPU 733.92 sec
2013-09-17 23:09:24,595 Stage-1 map = 100%, reduce = 68%, Cumulative CPU 733.92 sec
2013-09-17 23:09:25,601 Stage-1 map = 100%, reduce = 71%, Cumulative CPU 733.92 sec
2013-09-17 23:09:26,607 Stage-1 map = 100%, reduce = 71%, Cumulative CPU 733.92 sec
2013-09-17 23:09:27,613 Stage-1 map = 100%, reduce = 71%, Cumulative CPU 733.92 sec
2013-09-17 23:09:28,619 Stage-1 map = 100%, reduce = 75%, Cumulative CPU 733.92 sec
2013-09-17 23:09:29,625 Stage-1 map = 100%, reduce = 75%, Cumulative CPU 733.92 sec
2013-09-17 23:09:30,631 Stage-1 map = 100%, reduce = 75%, Cumulative CPU 733.92 sec
2013-09-17 23:09:31,637 Stage-1 map = 100%, reduce = 78%, Cumulative CPU 733.92 sec
2013-09-17 23:09:32,643 Stage-1 map = 100%, reduce = 78%, Cumulative CPU 733.92 sec
2013-09-17 23:09:33,650 Stage-1 map = 100%, reduce = 78%, Cumulative CPU 733.92 sec
2013-09-17 23:09:36,622 Stage-1 map = 100%, reduce = 81%, Cumulative CPU 733.92 sec
2013-09-17 23:09:37,628 Stage-1 map = 100%, reduce = 81%, Cumulative CPU 733.92 sec
2013-09-17 23:09:38,634 Stage-1 map = 100%, reduce = 81%, Cumulative CPU 733.92 sec
2013-09-17 23:09:39,648 Stage-1 map = 100%, reduce = 84%, Cumulative CPU 733.92 sec
2013-09-17 23:09:40,654 Stage-1 map = 100%, reduce = 84%, Cumulative CPU 733.92 sec
2013-09-17 23:09:41,660 Stage-1 map = 100%, reduce = 84%, Cumulative CPU 733.92 sec
2013-09-17 23:09:42,666 Stage-1 map = 100%, reduce = 88%, Cumulative CPU 733.92 sec
2013-09-17 23:09:43,672 Stage-1 map = 100%, reduce = 88%, Cumulative CPU 733.92 sec
2013-09-17 23:09:44,677 Stage-1 map = 100%, reduce = 88%, Cumulative CPU 733.92 sec
2013-09-17 23:09:45,683 Stage-1 map = 100%, reduce = 91%, Cumulative CPU 733.92 sec
2013-09-17 23:09:46,689 Stage-1 map = 100%, reduce = 91%, Cumulative CPU 733.92 sec
2013-09-17 23:09:47,695 Stage-1 map = 100%, reduce = 91%, Cumulative CPU 733.92 sec
2013-09-17 23:09:48,702 Stage-1 map = 100%, reduce = 94%, Cumulative CPU 733.92 sec
2013-09-17 23:09:49,708 Stage-1 map = 100%, reduce = 94%, Cumulative CPU 733.92 sec
2013-09-17 23:09:50,714 Stage-1 map = 100%, reduce = 94%, Cumulative CPU 733.92 sec
2013-09-17 23:09:51,720 Stage-1 map = 100%, reduce = 97%, Cumulative CPU 733.92 sec
2013-09-17 23:09:52,726 Stage-1 map = 100%, reduce = 97%, Cumulative CPU 733.92 sec
2013-09-17 23:09:53,732 Stage-1 map = 100%, reduce = 97%, Cumulative CPU 733.92 sec
2013-09-17 23:09:54,740 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 775.12 sec
2013-09-17 23:09:55,746 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 775.12 sec
MapReduce Total cumulative CPU time: 12 minutes 55 seconds 120 msec
Ended Job = job_201309172235_0016
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 1 Cumulative CPU: 775.12 sec HDFS Read: 663537300 HDFS Write: 9 SUCCESS
Total MapReduce CPU Time Spent: 12 minutes 55 seconds 120 msec
OK
17630976
Time taken: 110.69 seconds, Fetched: 1 row(s)
hive> quit;
times: 2
query: SELECT count(DISTINCT UserID) FROM hits_100m;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_22191@mturlrep13_201309172309_1602577058.txt
hive> SELECT count(DISTINCT UserID) FROM hits_100m;;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0017
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 1
2013-09-17 23:10:05,908 Stage-1 map = 0%, reduce = 0%
2013-09-17 23:10:18,029 Stage-1 map = 1%, reduce = 0%
2013-09-17 23:10:19,113 Stage-1 map = 16%, reduce = 0%
2013-09-17 23:10:20,139 Stage-1 map = 18%, reduce = 0%
2013-09-17 23:10:21,150 Stage-1 map = 19%, reduce = 0%
2013-09-17 23:10:22,158 Stage-1 map = 46%, reduce = 0%
2013-09-17 23:10:23,166 Stage-1 map = 56%, reduce = 0%
2013-09-17 23:10:24,186 Stage-1 map = 57%, reduce = 0%, Cumulative CPU 10.62 sec
2013-09-17 23:10:25,199 Stage-1 map = 63%, reduce = 0%, Cumulative CPU 10.62 sec
2013-09-17 23:10:26,206 Stage-1 map = 72%, reduce = 0%, Cumulative CPU 10.62 sec
2013-09-17 23:10:27,222 Stage-1 map = 75%, reduce = 0%, Cumulative CPU 37.91 sec
2013-09-17 23:10:28,230 Stage-1 map = 76%, reduce = 0%, Cumulative CPU 79.92 sec
2013-09-17 23:10:29,287 Stage-1 map = 80%, reduce = 0%, Cumulative CPU 172.39 sec
2013-09-17 23:10:30,300 Stage-1 map = 82%, reduce = 0%, Cumulative CPU 220.22 sec
2013-09-17 23:10:31,311 Stage-1 map = 86%, reduce = 0%, Cumulative CPU 326.41 sec
2013-09-17 23:10:32,317 Stage-1 map = 91%, reduce = 0%, Cumulative CPU 441.77 sec
2013-09-17 23:10:33,324 Stage-1 map = 92%, reduce = 0%, Cumulative CPU 481.76 sec
2013-09-17 23:10:34,330 Stage-1 map = 93%, reduce = 0%, Cumulative CPU 481.76 sec
2013-09-17 23:10:35,337 Stage-1 map = 96%, reduce = 0%, Cumulative CPU 481.76 sec
2013-09-17 23:10:36,344 Stage-1 map = 97%, reduce = 11%, Cumulative CPU 507.3 sec
2013-09-17 23:10:37,351 Stage-1 map = 98%, reduce = 11%, Cumulative CPU 610.94 sec
2013-09-17 23:10:38,357 Stage-1 map = 99%, reduce = 11%, Cumulative CPU 610.94 sec
2013-09-17 23:10:39,362 Stage-1 map = 100%, reduce = 26%, Cumulative CPU 693.79 sec
2013-09-17 23:10:40,368 Stage-1 map = 100%, reduce = 26%, Cumulative CPU 693.79 sec
2013-09-17 23:10:41,375 Stage-1 map = 100%, reduce = 26%, Cumulative CPU 693.79 sec
2013-09-17 23:10:42,381 Stage-1 map = 100%, reduce = 27%, Cumulative CPU 693.79 sec
2013-09-17 23:10:43,388 Stage-1 map = 100%, reduce = 27%, Cumulative CPU 693.79 sec
2013-09-17 23:10:44,393 Stage-1 map = 100%, reduce = 27%, Cumulative CPU 693.79 sec
2013-09-17 23:10:45,399 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 693.79 sec
2013-09-17 23:10:46,404 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 693.79 sec
2013-09-17 23:10:47,409 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 693.79 sec
2013-09-17 23:10:48,414 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 693.79 sec
2013-09-17 23:10:49,419 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 693.79 sec
2013-09-17 23:10:50,424 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 693.79 sec
2013-09-17 23:10:51,429 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 693.79 sec
2013-09-17 23:10:52,435 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 693.79 sec
2013-09-17 23:10:53,440 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 693.79 sec
2013-09-17 23:10:54,446 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 693.79 sec
2013-09-17 23:10:55,451 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 693.79 sec
2013-09-17 23:10:56,456 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 693.79 sec
2013-09-17 23:10:57,461 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 693.79 sec
2013-09-17 23:10:58,836 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 693.79 sec
2013-09-17 23:10:59,948 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 693.79 sec
2013-09-17 23:11:00,953 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 693.79 sec
2013-09-17 23:11:01,958 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 693.79 sec
2013-09-17 23:11:02,963 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 693.79 sec
2013-09-17 23:11:05,322 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 693.79 sec
2013-09-17 23:11:06,328 Stage-1 map = 100%, reduce = 67%, Cumulative CPU 693.79 sec
2013-09-17 23:11:07,333 Stage-1 map = 100%, reduce = 67%, Cumulative CPU 693.79 sec
2013-09-17 23:11:08,339 Stage-1 map = 100%, reduce = 67%, Cumulative CPU 693.79 sec
2013-09-17 23:11:09,344 Stage-1 map = 100%, reduce = 69%, Cumulative CPU 693.79 sec
2013-09-17 23:11:10,349 Stage-1 map = 100%, reduce = 69%, Cumulative CPU 693.79 sec
2013-09-17 23:11:12,853 Stage-1 map = 100%, reduce = 70%, Cumulative CPU 693.79 sec
2013-09-17 23:11:13,858 Stage-1 map = 100%, reduce = 70%, Cumulative CPU 693.79 sec
2013-09-17 23:11:14,864 Stage-1 map = 100%, reduce = 70%, Cumulative CPU 693.79 sec
2013-09-17 23:11:21,480 Stage-1 map = 100%, reduce = 73%, Cumulative CPU 734.95 sec
2013-09-17 23:11:22,485 Stage-1 map = 100%, reduce = 73%, Cumulative CPU 734.95 sec
2013-09-17 23:11:23,490 Stage-1 map = 100%, reduce = 73%, Cumulative CPU 734.95 sec
2013-09-17 23:11:24,496 Stage-1 map = 100%, reduce = 83%, Cumulative CPU 734.95 sec
2013-09-17 23:11:25,501 Stage-1 map = 100%, reduce = 83%, Cumulative CPU 734.95 sec
2013-09-17 23:11:26,506 Stage-1 map = 100%, reduce = 83%, Cumulative CPU 734.95 sec
2013-09-17 23:11:27,512 Stage-1 map = 100%, reduce = 86%, Cumulative CPU 734.95 sec
2013-09-17 23:11:28,517 Stage-1 map = 100%, reduce = 86%, Cumulative CPU 734.95 sec
2013-09-17 23:11:29,522 Stage-1 map = 100%, reduce = 86%, Cumulative CPU 734.95 sec
2013-09-17 23:11:30,527 Stage-1 map = 100%, reduce = 89%, Cumulative CPU 734.95 sec
2013-09-17 23:11:31,543 Stage-1 map = 100%, reduce = 89%, Cumulative CPU 734.95 sec
2013-09-17 23:11:32,548 Stage-1 map = 100%, reduce = 89%, Cumulative CPU 734.95 sec
2013-09-17 23:11:33,553 Stage-1 map = 100%, reduce = 92%, Cumulative CPU 734.95 sec
2013-09-17 23:11:34,558 Stage-1 map = 100%, reduce = 92%, Cumulative CPU 734.95 sec
2013-09-17 23:11:35,562 Stage-1 map = 100%, reduce = 92%, Cumulative CPU 734.95 sec
2013-09-17 23:11:36,568 Stage-1 map = 100%, reduce = 95%, Cumulative CPU 734.95 sec
2013-09-17 23:11:37,573 Stage-1 map = 100%, reduce = 95%, Cumulative CPU 734.95 sec
2013-09-17 23:11:38,578 Stage-1 map = 100%, reduce = 95%, Cumulative CPU 734.95 sec
2013-09-17 23:11:39,583 Stage-1 map = 100%, reduce = 98%, Cumulative CPU 734.95 sec
2013-09-17 23:11:40,588 Stage-1 map = 100%, reduce = 98%, Cumulative CPU 734.95 sec
2013-09-17 23:11:41,593 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 761.58 sec
2013-09-17 23:11:42,599 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 761.58 sec
2013-09-17 23:11:43,605 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 761.58 sec
MapReduce Total cumulative CPU time: 12 minutes 41 seconds 580 msec
Ended Job = job_201309172235_0017
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 1 Cumulative CPU: 761.58 sec HDFS Read: 663537300 HDFS Write: 9 SUCCESS
Total MapReduce CPU Time Spent: 12 minutes 41 seconds 580 msec
OK
17630976
Time taken: 105.953 seconds, Fetched: 1 row(s)
hive> quit;
times: 3
query: SELECT count(DISTINCT UserID) FROM hits_100m;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_28018@mturlrep13_201309172311_1489682234.txt
hive> SELECT count(DISTINCT UserID) FROM hits_100m;;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0018
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 1
2013-09-17 23:11:54,362 Stage-1 map = 0%, reduce = 0%
2013-09-17 23:12:06,510 Stage-1 map = 9%, reduce = 0%
2013-09-17 23:12:07,520 Stage-1 map = 20%, reduce = 0%
2013-09-17 23:12:08,525 Stage-1 map = 24%, reduce = 0%
2013-09-17 23:12:09,532 Stage-1 map = 35%, reduce = 0%
2013-09-17 23:12:10,537 Stage-1 map = 55%, reduce = 0%
2013-09-17 23:12:11,543 Stage-1 map = 58%, reduce = 0%
2013-09-17 23:12:12,555 Stage-1 map = 66%, reduce = 0%, Cumulative CPU 11.81 sec
2013-09-17 23:12:13,575 Stage-1 map = 72%, reduce = 0%, Cumulative CPU 11.81 sec
2013-09-17 23:12:14,581 Stage-1 map = 74%, reduce = 0%, Cumulative CPU 11.81 sec
2013-09-17 23:12:15,598 Stage-1 map = 76%, reduce = 0%, Cumulative CPU 81.59 sec
2013-09-17 23:12:16,636 Stage-1 map = 80%, reduce = 0%, Cumulative CPU 141.75 sec
2013-09-17 23:12:17,658 Stage-1 map = 83%, reduce = 0%, Cumulative CPU 223.95 sec
2013-09-17 23:12:18,665 Stage-1 map = 85%, reduce = 0%, Cumulative CPU 293.32 sec
2013-09-17 23:12:19,671 Stage-1 map = 91%, reduce = 0%, Cumulative CPU 427.68 sec
2013-09-17 23:12:20,678 Stage-1 map = 92%, reduce = 0%, Cumulative CPU 487.22 sec
2013-09-17 23:12:21,684 Stage-1 map = 93%, reduce = 0%, Cumulative CPU 642.1 sec
2013-09-17 23:12:22,691 Stage-1 map = 96%, reduce = 0%, Cumulative CPU 642.1 sec
2013-09-17 23:12:23,697 Stage-1 map = 96%, reduce = 11%, Cumulative CPU 642.1 sec
2013-09-17 23:12:24,703 Stage-1 map = 98%, reduce = 11%, Cumulative CPU 665.75 sec
2013-09-17 23:12:25,709 Stage-1 map = 99%, reduce = 11%, Cumulative CPU 680.97 sec
2013-09-17 23:12:26,715 Stage-1 map = 100%, reduce = 11%, Cumulative CPU 699.12 sec
2013-09-17 23:12:27,720 Stage-1 map = 100%, reduce = 26%, Cumulative CPU 699.12 sec
2013-09-17 23:12:28,725 Stage-1 map = 100%, reduce = 26%, Cumulative CPU 699.12 sec
2013-09-17 23:12:29,730 Stage-1 map = 100%, reduce = 26%, Cumulative CPU 699.12 sec
2013-09-17 23:12:30,736 Stage-1 map = 100%, reduce = 27%, Cumulative CPU 699.12 sec
2013-09-17 23:12:31,741 Stage-1 map = 100%, reduce = 27%, Cumulative CPU 699.12 sec
2013-09-17 23:12:32,747 Stage-1 map = 100%, reduce = 27%, Cumulative CPU 699.12 sec
2013-09-17 23:12:33,752 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 699.12 sec
2013-09-17 23:12:34,757 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 699.12 sec
2013-09-17 23:12:35,762 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 699.12 sec
2013-09-17 23:12:36,766 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 699.12 sec
2013-09-17 23:12:37,771 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 699.12 sec
2013-09-17 23:12:38,776 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 699.12 sec
2013-09-17 23:12:39,780 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 699.12 sec
2013-09-17 23:12:40,785 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 699.12 sec
2013-09-17 23:12:41,789 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 699.12 sec
2013-09-17 23:12:42,794 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 699.12 sec
2013-09-17 23:12:43,798 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 699.12 sec
2013-09-17 23:12:44,803 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 699.12 sec
2013-09-17 23:12:45,808 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 699.12 sec
2013-09-17 23:12:46,813 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 699.12 sec
2013-09-17 23:12:48,011 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 699.12 sec
2013-09-17 23:12:49,016 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 699.12 sec
2013-09-17 23:12:50,021 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 699.12 sec
2013-09-17 23:12:51,025 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 699.12 sec
2013-09-17 23:12:52,030 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 699.12 sec
2013-09-17 23:12:53,035 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 699.12 sec
2013-09-17 23:12:54,041 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 699.12 sec
2013-09-17 23:12:55,046 Stage-1 map = 100%, reduce = 67%, Cumulative CPU 699.12 sec
2013-09-17 23:12:56,051 Stage-1 map = 100%, reduce = 67%, Cumulative CPU 699.12 sec
2013-09-17 23:12:57,056 Stage-1 map = 100%, reduce = 67%, Cumulative CPU 699.12 sec
2013-09-17 23:12:58,061 Stage-1 map = 100%, reduce = 70%, Cumulative CPU 699.12 sec
2013-09-17 23:12:59,066 Stage-1 map = 100%, reduce = 70%, Cumulative CPU 699.12 sec
2013-09-17 23:13:00,071 Stage-1 map = 100%, reduce = 70%, Cumulative CPU 699.12 sec
2013-09-17 23:13:01,076 Stage-1 map = 100%, reduce = 73%, Cumulative CPU 699.12 sec
2013-09-17 23:13:02,081 Stage-1 map = 100%, reduce = 73%, Cumulative CPU 699.12 sec
2013-09-17 23:13:03,086 Stage-1 map = 100%, reduce = 73%, Cumulative CPU 699.12 sec
2013-09-17 23:13:09,756 Stage-1 map = 100%, reduce = 76%, Cumulative CPU 699.12 sec
2013-09-17 23:13:10,762 Stage-1 map = 100%, reduce = 76%, Cumulative CPU 699.12 sec
2013-09-17 23:13:11,767 Stage-1 map = 100%, reduce = 76%, Cumulative CPU 699.12 sec
2013-09-17 23:13:12,773 Stage-1 map = 100%, reduce = 80%, Cumulative CPU 699.12 sec
2013-09-17 23:13:13,779 Stage-1 map = 100%, reduce = 80%, Cumulative CPU 699.12 sec
2013-09-17 23:13:14,784 Stage-1 map = 100%, reduce = 80%, Cumulative CPU 699.12 sec
2013-09-17 23:13:15,790 Stage-1 map = 100%, reduce = 83%, Cumulative CPU 699.12 sec
2013-09-17 23:13:16,795 Stage-1 map = 100%, reduce = 83%, Cumulative CPU 699.12 sec
2013-09-17 23:13:17,800 Stage-1 map = 100%, reduce = 83%, Cumulative CPU 699.12 sec
2013-09-17 23:13:18,805 Stage-1 map = 100%, reduce = 86%, Cumulative CPU 699.12 sec
2013-09-17 23:13:19,815 Stage-1 map = 100%, reduce = 86%, Cumulative CPU 699.12 sec
2013-09-17 23:13:20,820 Stage-1 map = 100%, reduce = 86%, Cumulative CPU 699.12 sec
2013-09-17 23:13:21,827 Stage-1 map = 100%, reduce = 90%, Cumulative CPU 755.01 sec
2013-09-17 23:13:22,832 Stage-1 map = 100%, reduce = 90%, Cumulative CPU 755.01 sec
2013-09-17 23:13:23,837 Stage-1 map = 100%, reduce = 90%, Cumulative CPU 755.01 sec
2013-09-17 23:13:24,844 Stage-1 map = 100%, reduce = 93%, Cumulative CPU 755.01 sec
2013-09-17 23:13:25,849 Stage-1 map = 100%, reduce = 93%, Cumulative CPU 755.01 sec
2013-09-17 23:13:26,855 Stage-1 map = 100%, reduce = 93%, Cumulative CPU 755.01 sec
2013-09-17 23:13:27,860 Stage-1 map = 100%, reduce = 97%, Cumulative CPU 755.01 sec
2013-09-17 23:13:28,866 Stage-1 map = 100%, reduce = 97%, Cumulative CPU 755.01 sec
2013-09-17 23:13:29,872 Stage-1 map = 100%, reduce = 97%, Cumulative CPU 755.01 sec
2013-09-17 23:13:30,877 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 764.66 sec
2013-09-17 23:13:31,883 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 764.66 sec
2013-09-17 23:13:32,888 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 764.66 sec
MapReduce Total cumulative CPU time: 12 minutes 44 seconds 660 msec
Ended Job = job_201309172235_0018
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 1 Cumulative CPU: 764.66 sec HDFS Read: 663537300 HDFS Write: 9 SUCCESS
Total MapReduce CPU Time Spent: 12 minutes 44 seconds 660 msec
OK
17630976
Time taken: 107.343 seconds, Fetched: 1 row(s)
hive> quit;
times: 1
query: SELECT count(DISTINCT SearchPhrase) FROM hits_100m;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_709@mturlrep13_201309172313_1257295877.txt
hive> SELECT count(DISTINCT SearchPhrase) FROM hits_100m;;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0019
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 1
2013-09-17 23:13:50,358 Stage-1 map = 0%, reduce = 0%
2013-09-17 23:14:00,485 Stage-1 map = 1%, reduce = 0%
2013-09-17 23:14:01,491 Stage-1 map = 2%, reduce = 0%
2013-09-17 23:14:02,497 Stage-1 map = 11%, reduce = 0%
2013-09-17 23:14:04,042 Stage-1 map = 24%, reduce = 0%
2013-09-17 23:14:05,047 Stage-1 map = 26%, reduce = 0%
2013-09-17 23:14:06,057 Stage-1 map = 35%, reduce = 0%
2013-09-17 23:14:07,069 Stage-1 map = 55%, reduce = 0%, Cumulative CPU 9.77 sec
2013-09-17 23:14:08,077 Stage-1 map = 55%, reduce = 0%, Cumulative CPU 9.77 sec
2013-09-17 23:14:09,260 Stage-1 map = 64%, reduce = 0%, Cumulative CPU 9.77 sec
2013-09-17 23:14:10,285 Stage-1 map = 79%, reduce = 0%, Cumulative CPU 54.19 sec
2013-09-17 23:14:11,298 Stage-1 map = 82%, reduce = 0%, Cumulative CPU 98.66 sec
2013-09-17 23:14:12,311 Stage-1 map = 93%, reduce = 0%, Cumulative CPU 229.9 sec
2013-09-17 23:14:13,317 Stage-1 map = 97%, reduce = 0%, Cumulative CPU 354.88 sec
2013-09-17 23:14:14,323 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 470.21 sec
2013-09-17 23:14:15,329 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 470.21 sec
2013-09-17 23:14:16,334 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 470.21 sec
2013-09-17 23:14:17,340 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 470.21 sec
2013-09-17 23:14:18,346 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 470.21 sec
2013-09-17 23:14:19,353 Stage-1 map = 100%, reduce = 8%, Cumulative CPU 470.21 sec
2013-09-17 23:14:20,359 Stage-1 map = 100%, reduce = 8%, Cumulative CPU 470.21 sec
2013-09-17 23:14:21,365 Stage-1 map = 100%, reduce = 8%, Cumulative CPU 470.21 sec
2013-09-17 23:14:22,370 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 477.68 sec
2013-09-17 23:14:23,376 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 477.68 sec
2013-09-17 23:14:24,382 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 477.68 sec
2013-09-17 23:14:25,387 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 477.68 sec
2013-09-17 23:14:26,393 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 477.68 sec
2013-09-17 23:14:27,399 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 477.68 sec
2013-09-17 23:14:28,404 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 477.68 sec
2013-09-17 23:14:29,410 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 477.68 sec
2013-09-17 23:14:30,416 Stage-1 map = 100%, reduce = 68%, Cumulative CPU 477.68 sec
2013-09-17 23:14:31,421 Stage-1 map = 100%, reduce = 68%, Cumulative CPU 477.68 sec
2013-09-17 23:14:32,425 Stage-1 map = 100%, reduce = 68%, Cumulative CPU 477.68 sec
2013-09-17 23:14:33,430 Stage-1 map = 100%, reduce = 68%, Cumulative CPU 477.68 sec
2013-09-17 23:14:34,436 Stage-1 map = 100%, reduce = 75%, Cumulative CPU 477.68 sec
2013-09-17 23:14:35,440 Stage-1 map = 100%, reduce = 75%, Cumulative CPU 477.68 sec
2013-09-17 23:14:36,446 Stage-1 map = 100%, reduce = 75%, Cumulative CPU 477.68 sec
2013-09-17 23:14:37,451 Stage-1 map = 100%, reduce = 82%, Cumulative CPU 477.68 sec
2013-09-17 23:14:38,456 Stage-1 map = 100%, reduce = 82%, Cumulative CPU 477.68 sec
2013-09-17 23:14:39,479 Stage-1 map = 100%, reduce = 90%, Cumulative CPU 477.68 sec
2013-09-17 23:14:40,484 Stage-1 map = 100%, reduce = 90%, Cumulative CPU 477.68 sec
2013-09-17 23:14:41,489 Stage-1 map = 100%, reduce = 90%, Cumulative CPU 477.68 sec
2013-09-17 23:14:42,494 Stage-1 map = 100%, reduce = 90%, Cumulative CPU 477.68 sec
2013-09-17 23:14:43,499 Stage-1 map = 100%, reduce = 97%, Cumulative CPU 477.68 sec
2013-09-17 23:14:44,506 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 501.74 sec
2013-09-17 23:14:45,512 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 501.74 sec
MapReduce Total cumulative CPU time: 8 minutes 21 seconds 740 msec
Ended Job = job_201309172235_0019
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 1 Cumulative CPU: 501.74 sec HDFS Read: 182281719 HDFS Write: 8 SUCCESS
Total MapReduce CPU Time Spent: 8 minutes 21 seconds 740 msec
OK
6404441
Time taken: 68.119 seconds, Fetched: 1 row(s)
hive> quit;
times: 2
query: SELECT count(DISTINCT SearchPhrase) FROM hits_100m;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_5905@mturlrep13_201309172314_300988988.txt
hive> SELECT count(DISTINCT SearchPhrase) FROM hits_100m;;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0020
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 1
2013-09-17 23:14:57,829 Stage-1 map = 0%, reduce = 0%
2013-09-17 23:15:09,948 Stage-1 map = 1%, reduce = 0%
2013-09-17 23:15:10,963 Stage-1 map = 17%, reduce = 0%
2013-09-17 23:15:11,975 Stage-1 map = 37%, reduce = 0%
2013-09-17 23:15:12,995 Stage-1 map = 38%, reduce = 0%
2013-09-17 23:15:14,001 Stage-1 map = 61%, reduce = 0%
2013-09-17 23:15:15,016 Stage-1 map = 84%, reduce = 0%, Cumulative CPU 20.56 sec
2013-09-17 23:15:16,039 Stage-1 map = 87%, reduce = 0%, Cumulative CPU 120.41 sec
2013-09-17 23:15:17,051 Stage-1 map = 92%, reduce = 0%, Cumulative CPU 219.11 sec
2013-09-17 23:15:18,058 Stage-1 map = 95%, reduce = 0%, Cumulative CPU 287.4 sec
2013-09-17 23:15:19,064 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 475.43 sec
2013-09-17 23:15:20,070 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 475.43 sec
2013-09-17 23:15:21,075 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 475.43 sec
2013-09-17 23:15:22,081 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 475.43 sec
2013-09-17 23:15:23,087 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 475.43 sec
2013-09-17 23:15:24,093 Stage-1 map = 100%, reduce = 17%, Cumulative CPU 475.43 sec
2013-09-17 23:15:25,099 Stage-1 map = 100%, reduce = 17%, Cumulative CPU 475.43 sec
2013-09-17 23:15:26,104 Stage-1 map = 100%, reduce = 17%, Cumulative CPU 475.43 sec
2013-09-17 23:15:27,109 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 475.43 sec
2013-09-17 23:15:28,115 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 475.43 sec
2013-09-17 23:15:29,120 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 475.43 sec
2013-09-17 23:15:30,126 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 475.43 sec
2013-09-17 23:15:31,132 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 475.43 sec
2013-09-17 23:15:32,138 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 475.43 sec
2013-09-17 23:15:33,144 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 475.43 sec
2013-09-17 23:15:34,150 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 475.43 sec
2013-09-17 23:15:35,156 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 475.43 sec
2013-09-17 23:15:36,161 Stage-1 map = 100%, reduce = 68%, Cumulative CPU 475.43 sec
2013-09-17 23:15:37,167 Stage-1 map = 100%, reduce = 68%, Cumulative CPU 475.43 sec
2013-09-17 23:15:38,172 Stage-1 map = 100%, reduce = 68%, Cumulative CPU 475.43 sec
2013-09-17 23:15:39,177 Stage-1 map = 100%, reduce = 74%, Cumulative CPU 475.43 sec
2013-09-17 23:15:40,182 Stage-1 map = 100%, reduce = 74%, Cumulative CPU 475.43 sec
2013-09-17 23:15:41,187 Stage-1 map = 100%, reduce = 74%, Cumulative CPU 475.43 sec
2013-09-17 23:15:42,192 Stage-1 map = 100%, reduce = 81%, Cumulative CPU 475.43 sec
2013-09-17 23:15:43,198 Stage-1 map = 100%, reduce = 81%, Cumulative CPU 475.43 sec
2013-09-17 23:15:44,203 Stage-1 map = 100%, reduce = 81%, Cumulative CPU 475.43 sec
2013-09-17 23:15:45,208 Stage-1 map = 100%, reduce = 88%, Cumulative CPU 475.43 sec
2013-09-17 23:15:46,214 Stage-1 map = 100%, reduce = 88%, Cumulative CPU 475.43 sec
2013-09-17 23:15:47,219 Stage-1 map = 100%, reduce = 88%, Cumulative CPU 475.43 sec
2013-09-17 23:15:48,224 Stage-1 map = 100%, reduce = 95%, Cumulative CPU 475.43 sec
2013-09-17 23:15:49,229 Stage-1 map = 100%, reduce = 95%, Cumulative CPU 475.43 sec
2013-09-17 23:15:50,237 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 508.08 sec
2013-09-17 23:15:53,774 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 508.08 sec
2013-09-17 23:15:54,834 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 508.08 sec
MapReduce Total cumulative CPU time: 8 minutes 28 seconds 80 msec
Ended Job = job_201309172235_0020
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 1 Cumulative CPU: 508.08 sec HDFS Read: 182281719 HDFS Write: 8 SUCCESS
Total MapReduce CPU Time Spent: 8 minutes 28 seconds 80 msec
OK
6404441
Time taken: 64.831 seconds, Fetched: 1 row(s)
hive> quit;
times: 3
query: SELECT count(DISTINCT SearchPhrase) FROM hits_100m;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_11972@mturlrep13_201309172315_1325234581.txt
hive> SELECT count(DISTINCT SearchPhrase) FROM hits_100m;;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0021
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 1
2013-09-17 23:16:04,362 Stage-1 map = 0%, reduce = 0%
2013-09-17 23:16:16,459 Stage-1 map = 2%, reduce = 0%
2013-09-17 23:16:17,471 Stage-1 map = 17%, reduce = 0%
2013-09-17 23:16:18,484 Stage-1 map = 31%, reduce = 0%
2013-09-17 23:16:19,512 Stage-1 map = 36%, reduce = 0%
2013-09-17 23:16:20,525 Stage-1 map = 66%, reduce = 0%
2013-09-17 23:16:21,542 Stage-1 map = 77%, reduce = 0%, Cumulative CPU 31.11 sec
2013-09-17 23:16:22,565 Stage-1 map = 86%, reduce = 0%, Cumulative CPU 364.53 sec
2013-09-17 23:16:23,583 Stage-1 map = 93%, reduce = 0%, Cumulative CPU 382.31 sec
2013-09-17 23:16:24,593 Stage-1 map = 97%, reduce = 0%, Cumulative CPU 432.98 sec
2013-09-17 23:16:25,599 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 472.89 sec
2013-09-17 23:16:26,604 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 472.89 sec
2013-09-17 23:16:27,610 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 472.89 sec
2013-09-17 23:16:28,615 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 472.89 sec
2013-09-17 23:16:29,621 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 472.89 sec
2013-09-17 23:16:30,626 Stage-1 map = 100%, reduce = 16%, Cumulative CPU 472.89 sec
2013-09-17 23:16:31,632 Stage-1 map = 100%, reduce = 16%, Cumulative CPU 472.89 sec
2013-09-17 23:16:32,638 Stage-1 map = 100%, reduce = 16%, Cumulative CPU 472.89 sec
2013-09-17 23:16:33,643 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 472.89 sec
2013-09-17 23:16:34,649 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 472.89 sec
2013-09-17 23:16:35,654 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 472.89 sec
2013-09-17 23:16:36,660 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 472.89 sec
2013-09-17 23:16:37,665 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 472.89 sec
2013-09-17 23:16:38,671 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 472.89 sec
2013-09-17 23:16:39,677 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 472.89 sec
2013-09-17 23:16:40,682 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 472.89 sec
2013-09-17 23:16:41,688 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 472.89 sec
2013-09-17 23:16:42,694 Stage-1 map = 100%, reduce = 69%, Cumulative CPU 472.89 sec
2013-09-17 23:16:43,699 Stage-1 map = 100%, reduce = 69%, Cumulative CPU 472.89 sec
2013-09-17 23:16:44,704 Stage-1 map = 100%, reduce = 69%, Cumulative CPU 472.89 sec
2013-09-17 23:16:45,710 Stage-1 map = 100%, reduce = 75%, Cumulative CPU 472.89 sec
2013-09-17 23:16:46,715 Stage-1 map = 100%, reduce = 75%, Cumulative CPU 472.89 sec
2013-09-17 23:16:47,721 Stage-1 map = 100%, reduce = 75%, Cumulative CPU 472.89 sec
2013-09-17 23:16:48,726 Stage-1 map = 100%, reduce = 82%, Cumulative CPU 472.89 sec
2013-09-17 23:16:49,731 Stage-1 map = 100%, reduce = 82%, Cumulative CPU 472.89 sec
2013-09-17 23:16:50,736 Stage-1 map = 100%, reduce = 82%, Cumulative CPU 472.89 sec
2013-09-17 23:16:51,741 Stage-1 map = 100%, reduce = 89%, Cumulative CPU 472.89 sec
2013-09-17 23:16:52,746 Stage-1 map = 100%, reduce = 89%, Cumulative CPU 472.89 sec
2013-09-17 23:16:53,751 Stage-1 map = 100%, reduce = 89%, Cumulative CPU 472.89 sec
2013-09-17 23:16:54,770 Stage-1 map = 100%, reduce = 97%, Cumulative CPU 472.89 sec
2013-09-17 23:16:55,775 Stage-1 map = 100%, reduce = 97%, Cumulative CPU 472.89 sec
2013-09-17 23:16:56,783 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 504.75 sec
2013-09-17 23:16:57,788 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 504.75 sec
MapReduce Total cumulative CPU time: 8 minutes 24 seconds 750 msec
Ended Job = job_201309172235_0021
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 1 Cumulative CPU: 504.75 sec HDFS Read: 182281719 HDFS Write: 8 SUCCESS
Total MapReduce CPU Time Spent: 8 minutes 24 seconds 750 msec
OK
6404441
Time taken: 64.269 seconds, Fetched: 1 row(s)
hive> quit;
times: 1
query: SELECT min(EventDate), max(EventDate) FROM hits_100m;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_17050@mturlrep13_201309172317_1762378997.txt
hive> SELECT min(EventDate), max(EventDate) FROM hits_100m;;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0022
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 1
2013-09-17 23:17:18,068 Stage-1 map = 0%, reduce = 0%
2013-09-17 23:17:30,186 Stage-1 map = 4%, reduce = 0%
2013-09-17 23:17:31,192 Stage-1 map = 30%, reduce = 0%
2013-09-17 23:17:32,197 Stage-1 map = 31%, reduce = 0%
2013-09-17 23:17:33,206 Stage-1 map = 36%, reduce = 0%
2013-09-17 23:17:34,211 Stage-1 map = 71%, reduce = 0%
2013-09-17 23:17:35,223 Stage-1 map = 76%, reduce = 0%, Cumulative CPU 50.55 sec
2013-09-17 23:17:36,231 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 318.5 sec
2013-09-17 23:17:37,237 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 318.5 sec
2013-09-17 23:17:38,242 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 318.5 sec
2013-09-17 23:17:39,248 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 318.5 sec
2013-09-17 23:17:40,253 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 318.5 sec
2013-09-17 23:17:41,259 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 318.5 sec
2013-09-17 23:17:42,265 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 318.5 sec
2013-09-17 23:17:43,271 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 318.5 sec
2013-09-17 23:17:44,278 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 321.17 sec
2013-09-17 23:17:45,284 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 321.17 sec
MapReduce Total cumulative CPU time: 5 minutes 21 seconds 170 msec
Ended Job = job_201309172235_0022
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 1 Cumulative CPU: 321.17 sec HDFS Read: 4951094 HDFS Write: 6 SUCCESS
Total MapReduce CPU Time Spent: 5 minutes 21 seconds 170 msec
OK
Time taken: 37.809 seconds, Fetched: 1 row(s)
hive> quit;
times: 2
query: SELECT min(EventDate), max(EventDate) FROM hits_100m;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_22045@mturlrep13_201309172317_1976013998.txt
hive> SELECT min(EventDate), max(EventDate) FROM hits_100m;;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0023
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 1
2013-09-17 23:17:55,984 Stage-1 map = 0%, reduce = 0%
2013-09-17 23:18:07,051 Stage-1 map = 2%, reduce = 0%
2013-09-17 23:18:08,056 Stage-1 map = 37%, reduce = 0%
2013-09-17 23:18:09,084 Stage-1 map = 55%, reduce = 0%, Cumulative CPU 15.31 sec
2013-09-17 23:18:10,092 Stage-1 map = 83%, reduce = 0%, Cumulative CPU 213.11 sec
2013-09-17 23:18:11,100 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 322.89 sec
2013-09-17 23:18:12,105 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 322.89 sec
2013-09-17 23:18:13,111 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 322.89 sec
2013-09-17 23:18:14,117 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 322.89 sec
2013-09-17 23:18:15,123 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 322.89 sec
2013-09-17 23:18:16,128 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 322.89 sec
2013-09-17 23:18:17,134 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 322.89 sec
2013-09-17 23:18:18,141 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 325.69 sec
2013-09-17 23:18:19,147 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 325.69 sec
2013-09-17 23:18:20,153 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 325.69 sec
MapReduce Total cumulative CPU time: 5 minutes 25 seconds 690 msec
Ended Job = job_201309172235_0023
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 1 Cumulative CPU: 325.69 sec HDFS Read: 4951094 HDFS Write: 6 SUCCESS
Total MapReduce CPU Time Spent: 5 minutes 25 seconds 690 msec
OK
Time taken: 33.021 seconds, Fetched: 1 row(s)
hive> quit;
times: 3
query: SELECT min(EventDate), max(EventDate) FROM hits_100m;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_26986@mturlrep13_201309172318_1806287056.txt
hive> SELECT min(EventDate), max(EventDate) FROM hits_100m;;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0024
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 1
2013-09-17 23:18:30,852 Stage-1 map = 0%, reduce = 0%
2013-09-17 23:18:42,000 Stage-1 map = 1%, reduce = 0%
2013-09-17 23:18:43,008 Stage-1 map = 28%, reduce = 0%
2013-09-17 23:18:44,021 Stage-1 map = 57%, reduce = 0%, Cumulative CPU 41.29 sec
2013-09-17 23:18:45,035 Stage-1 map = 88%, reduce = 0%, Cumulative CPU 247.75 sec
2013-09-17 23:18:46,043 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 322.15 sec
2013-09-17 23:18:47,048 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 322.15 sec
2013-09-17 23:18:48,053 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 322.15 sec
2013-09-17 23:18:49,059 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 322.15 sec
2013-09-17 23:18:50,064 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 322.15 sec
2013-09-17 23:18:51,070 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 322.15 sec
2013-09-17 23:18:52,076 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 322.15 sec
2013-09-17 23:18:53,081 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 322.15 sec
2013-09-17 23:18:54,089 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 325.35 sec
2013-09-17 23:18:55,095 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 325.35 sec
MapReduce Total cumulative CPU time: 5 minutes 25 seconds 350 msec
Ended Job = job_201309172235_0024
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 1 Cumulative CPU: 325.35 sec HDFS Read: 4951094 HDFS Write: 6 SUCCESS
Total MapReduce CPU Time Spent: 5 minutes 25 seconds 350 msec
OK
Time taken: 33.13 seconds, Fetched: 1 row(s)
hive> quit;
times: 1
query: SELECT AdvEngineID, count(*) AS c FROM hits_100m WHERE AdvEngineID != 0 GROUP BY AdvEngineID ORDER BY c DESC;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_31945@mturlrep13_201309172319_144240708.txt
hive> SELECT AdvEngineID, count(*) AS c FROM hits_100m WHERE AdvEngineID != 0 GROUP BY AdvEngineID ORDER BY c DESC;;
Total MapReduce jobs = 2
Launching Job 1 out of 2
Number of reduce tasks not specified. Estimated from input data size: 11
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0025
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 11
2013-09-17 23:19:12,376 Stage-1 map = 0%, reduce = 0%
2013-09-17 23:19:22,455 Stage-1 map = 1%, reduce = 0%, Cumulative CPU 53.86 sec
2013-09-17 23:19:23,463 Stage-1 map = 1%, reduce = 0%, Cumulative CPU 53.86 sec
2013-09-17 23:19:24,470 Stage-1 map = 13%, reduce = 0%, Cumulative CPU 53.86 sec
2013-09-17 23:19:25,992 Stage-1 map = 36%, reduce = 0%, Cumulative CPU 55.65 sec
2013-09-17 23:19:27,002 Stage-1 map = 37%, reduce = 0%, Cumulative CPU 55.65 sec
2013-09-17 23:19:28,010 Stage-1 map = 57%, reduce = 0%, Cumulative CPU 97.8 sec
2013-09-17 23:19:29,023 Stage-1 map = 89%, reduce = 0%, Cumulative CPU 216.67 sec
2013-09-17 23:19:30,033 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 288.99 sec
2013-09-17 23:19:31,040 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 288.99 sec
2013-09-17 23:19:32,046 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 288.99 sec
2013-09-17 23:19:33,052 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 288.99 sec
2013-09-17 23:19:34,058 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 288.99 sec
2013-09-17 23:19:35,065 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 288.99 sec
2013-09-17 23:19:36,070 Stage-1 map = 100%, reduce = 3%, Cumulative CPU 288.99 sec
2013-09-17 23:19:37,080 Stage-1 map = 100%, reduce = 30%, Cumulative CPU 292.1 sec
2013-09-17 23:19:38,087 Stage-1 map = 100%, reduce = 70%, Cumulative CPU 310.42 sec
2013-09-17 23:19:39,094 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 328.04 sec
2013-09-17 23:19:40,101 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 328.04 sec
2013-09-17 23:19:41,111 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 328.04 sec
MapReduce Total cumulative CPU time: 5 minutes 28 seconds 40 msec
Ended Job = job_201309172235_0025
Launching Job 2 out of 2
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0026
Hadoop job information for Stage-2: number of mappers: 1; number of reducers: 1
2013-09-17 23:19:43,669 Stage-2 map = 0%, reduce = 0%
2013-09-17 23:19:45,678 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.75 sec
2013-09-17 23:19:46,683 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.75 sec
2013-09-17 23:19:47,689 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.75 sec
2013-09-17 23:19:48,694 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.75 sec
2013-09-17 23:19:49,698 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.75 sec
2013-09-17 23:19:50,703 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.75 sec
2013-09-17 23:19:51,709 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.75 sec
2013-09-17 23:19:52,715 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 0.75 sec
2013-09-17 23:19:53,721 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 2.37 sec
2013-09-17 23:19:54,727 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 2.37 sec
2013-09-17 23:19:55,733 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 2.37 sec
MapReduce Total cumulative CPU time: 2 seconds 370 msec
Ended Job = job_201309172235_0026
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 11 Cumulative CPU: 328.04 sec HDFS Read: 6282387 HDFS Write: 1443 SUCCESS
Job 1: Map: 1 Reduce: 1 Cumulative CPU: 2.37 sec HDFS Read: 5020 HDFS Write: 129 SUCCESS
Total MapReduce CPU Time Spent: 5 minutes 30 seconds 410 msec
OK
Time taken: 53.788 seconds, Fetched: 18 row(s)
hive> quit;
times: 2
query: SELECT AdvEngineID, count(*) AS c FROM hits_100m WHERE AdvEngineID != 0 GROUP BY AdvEngineID ORDER BY c DESC;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_6554@mturlrep13_201309172319_1947847771.txt
hive> SELECT AdvEngineID, count(*) AS c FROM hits_100m WHERE AdvEngineID != 0 GROUP BY AdvEngineID ORDER BY c DESC;;
Total MapReduce jobs = 2
Launching Job 1 out of 2
Number of reduce tasks not specified. Estimated from input data size: 11
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0027
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 11
2013-09-17 23:20:07,904 Stage-1 map = 0%, reduce = 0%
2013-09-17 23:20:19,166 Stage-1 map = 4%, reduce = 0%
2013-09-17 23:20:20,189 Stage-1 map = 42%, reduce = 0%
2013-09-17 23:20:21,209 Stage-1 map = 79%, reduce = 0%, Cumulative CPU 147.41 sec
2013-09-17 23:20:22,217 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 296.25 sec
2013-09-17 23:20:23,226 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 296.25 sec
2013-09-17 23:20:24,237 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 296.25 sec
2013-09-17 23:20:25,244 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 296.25 sec
2013-09-17 23:20:26,251 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 296.25 sec
2013-09-17 23:20:27,257 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 296.25 sec
2013-09-17 23:20:28,264 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 296.25 sec
2013-09-17 23:20:29,270 Stage-1 map = 100%, reduce = 12%, Cumulative CPU 296.25 sec
2013-09-17 23:20:30,279 Stage-1 map = 100%, reduce = 39%, Cumulative CPU 305.77 sec
2013-09-17 23:20:31,286 Stage-1 map = 100%, reduce = 67%, Cumulative CPU 315.54 sec
2013-09-17 23:20:32,293 Stage-1 map = 100%, reduce = 94%, Cumulative CPU 327.93 sec
2013-09-17 23:20:33,300 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 331.3 sec
2013-09-17 23:20:34,306 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 331.3 sec
MapReduce Total cumulative CPU time: 5 minutes 31 seconds 300 msec
Ended Job = job_201309172235_0027
Launching Job 2 out of 2
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0028
Hadoop job information for Stage-2: number of mappers: 1; number of reducers: 1
2013-09-17 23:20:36,785 Stage-2 map = 0%, reduce = 0%
2013-09-17 23:20:38,794 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.72 sec
2013-09-17 23:20:39,799 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.72 sec
2013-09-17 23:20:40,803 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.72 sec
2013-09-17 23:20:41,808 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.72 sec
2013-09-17 23:20:42,812 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.72 sec
2013-09-17 23:20:43,817 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.72 sec
2013-09-17 23:20:44,821 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.72 sec
2013-09-17 23:20:45,826 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 0.72 sec
2013-09-17 23:20:46,831 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 2.34 sec
2013-09-17 23:20:47,837 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 2.34 sec
2013-09-17 23:20:48,842 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 2.34 sec
MapReduce Total cumulative CPU time: 2 seconds 340 msec
Ended Job = job_201309172235_0028
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 11 Cumulative CPU: 331.3 sec HDFS Read: 6282387 HDFS Write: 1443 SUCCESS
Job 1: Map: 1 Reduce: 1 Cumulative CPU: 2.34 sec HDFS Read: 5020 HDFS Write: 129 SUCCESS
Total MapReduce CPU Time Spent: 5 minutes 33 seconds 640 msec
OK
Time taken: 51.261 seconds, Fetched: 18 row(s)
hive> quit;
times: 3
query: SELECT AdvEngineID, count(*) AS c FROM hits_100m WHERE AdvEngineID != 0 GROUP BY AdvEngineID ORDER BY c DESC;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_13828@mturlrep13_201309172320_1002752164.txt
hive> SELECT AdvEngineID, count(*) AS c FROM hits_100m WHERE AdvEngineID != 0 GROUP BY AdvEngineID ORDER BY c DESC;;
Total MapReduce jobs = 2
Launching Job 1 out of 2
Number of reduce tasks not specified. Estimated from input data size: 11
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0029
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 11
2013-09-17 23:20:59,464 Stage-1 map = 0%, reduce = 0%
2013-09-17 23:21:08,619 Stage-1 map = 1%, reduce = 0%
2013-09-17 23:21:10,697 Stage-1 map = 4%, reduce = 0%, Cumulative CPU 7.3 sec
2013-09-17 23:21:11,718 Stage-1 map = 61%, reduce = 0%, Cumulative CPU 28.78 sec
2013-09-17 23:21:12,731 Stage-1 map = 95%, reduce = 0%, Cumulative CPU 254.07 sec
2013-09-17 23:21:13,739 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 287.86 sec
2013-09-17 23:21:14,748 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 287.86 sec
2013-09-17 23:21:15,755 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 287.86 sec
2013-09-17 23:21:16,762 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 287.86 sec
2013-09-17 23:21:17,769 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 287.86 sec
2013-09-17 23:21:18,776 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 287.86 sec
2013-09-17 23:21:19,782 Stage-1 map = 100%, reduce = 6%, Cumulative CPU 287.86 sec
2013-09-17 23:21:20,789 Stage-1 map = 100%, reduce = 18%, Cumulative CPU 287.86 sec
2013-09-17 23:21:21,798 Stage-1 map = 100%, reduce = 67%, Cumulative CPU 307.51 sec
2013-09-17 23:21:22,804 Stage-1 map = 100%, reduce = 88%, Cumulative CPU 319.28 sec
2013-09-17 23:21:23,811 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 323.91 sec
2013-09-17 23:21:24,817 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 323.91 sec
MapReduce Total cumulative CPU time: 5 minutes 23 seconds 910 msec
Ended Job = job_201309172235_0029
Launching Job 2 out of 2
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0030
Hadoop job information for Stage-2: number of mappers: 1; number of reducers: 1
2013-09-17 23:21:28,286 Stage-2 map = 0%, reduce = 0%
2013-09-17 23:21:29,293 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.69 sec
2013-09-17 23:21:30,299 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.69 sec
2013-09-17 23:21:31,304 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.69 sec
2013-09-17 23:21:32,309 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.69 sec
2013-09-17 23:21:33,314 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.69 sec
2013-09-17 23:21:34,320 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.69 sec
2013-09-17 23:21:35,325 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.69 sec
2013-09-17 23:21:36,330 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.69 sec
2013-09-17 23:21:37,336 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 2.13 sec
2013-09-17 23:21:38,341 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 2.13 sec
2013-09-17 23:21:39,348 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 2.13 sec
MapReduce Total cumulative CPU time: 2 seconds 130 msec
Ended Job = job_201309172235_0030
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 11 Cumulative CPU: 323.91 sec HDFS Read: 6282387 HDFS Write: 1443 SUCCESS
Job 1: Map: 1 Reduce: 1 Cumulative CPU: 2.13 sec HDFS Read: 5009 HDFS Write: 129 SUCCESS
Total MapReduce CPU Time Spent: 5 minutes 26 seconds 40 msec
OK
Time taken: 48.653 seconds, Fetched: 18 row(s)
hive> quit;
-- мощная фильтрация. После фильтрации почти ничего не остаётся, но делаем ещё агрегацию.;
times: 1
query: SELECT RegionID, count(DISTINCT UserID) AS u FROM hits_100m GROUP BY RegionID ORDER BY u DESC LIMIT 10;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_20466@mturlrep13_201309172321_1193168986.txt
hive> SELECT RegionID, count(DISTINCT UserID) AS u FROM hits_100m GROUP BY RegionID ORDER BY u DESC LIMIT 10;;
Total MapReduce jobs = 2
Launching Job 1 out of 2
Number of reduce tasks not specified. Estimated from input data size: 11
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0031
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 11
2013-09-17 23:21:56,900 Stage-1 map = 0%, reduce = 0%
2013-09-17 23:22:07,016 Stage-1 map = 1%, reduce = 0%
2013-09-17 23:22:08,024 Stage-1 map = 3%, reduce = 0%
2013-09-17 23:22:09,036 Stage-1 map = 4%, reduce = 0%
2013-09-17 23:22:10,195 Stage-1 map = 17%, reduce = 0%
2013-09-17 23:22:11,201 Stage-1 map = 18%, reduce = 0%
2013-09-17 23:22:12,207 Stage-1 map = 24%, reduce = 0%
2013-09-17 23:22:13,212 Stage-1 map = 32%, reduce = 0%
2013-09-17 23:22:14,220 Stage-1 map = 36%, reduce = 0%
2013-09-17 23:22:16,775 Stage-1 map = 52%, reduce = 0%
2013-09-17 23:22:18,821 Stage-1 map = 53%, reduce = 0%, Cumulative CPU 39.61 sec
2013-09-17 23:22:19,842 Stage-1 map = 73%, reduce = 0%, Cumulative CPU 39.61 sec
2013-09-17 23:22:20,847 Stage-1 map = 75%, reduce = 0%, Cumulative CPU 39.61 sec
2013-09-17 23:22:21,854 Stage-1 map = 75%, reduce = 0%, Cumulative CPU 59.18 sec
2013-09-17 23:22:22,860 Stage-1 map = 80%, reduce = 0%, Cumulative CPU 538.37 sec
2013-09-17 23:22:23,900 Stage-1 map = 81%, reduce = 0%, Cumulative CPU 540.67 sec
2013-09-17 23:22:25,023 Stage-1 map = 81%, reduce = 0%, Cumulative CPU 541.84 sec
2013-09-17 23:22:26,034 Stage-1 map = 86%, reduce = 0%, Cumulative CPU 571.25 sec
2013-09-17 23:22:27,044 Stage-1 map = 88%, reduce = 0%, Cumulative CPU 595.16 sec
2013-09-17 23:22:28,050 Stage-1 map = 89%, reduce = 0%, Cumulative CPU 618.83 sec
2013-09-17 23:22:29,057 Stage-1 map = 95%, reduce = 0%, Cumulative CPU 647.47 sec
2013-09-17 23:22:30,063 Stage-1 map = 96%, reduce = 0%, Cumulative CPU 664.76 sec
2013-09-17 23:22:31,070 Stage-1 map = 96%, reduce = 1%, Cumulative CPU 664.76 sec
2013-09-17 23:22:32,076 Stage-1 map = 98%, reduce = 12%, Cumulative CPU 685.09 sec
2013-09-17 23:22:33,082 Stage-1 map = 99%, reduce = 12%, Cumulative CPU 710.3 sec
2013-09-17 23:22:34,088 Stage-1 map = 99%, reduce = 18%, Cumulative CPU 722.47 sec
2013-09-17 23:22:35,094 Stage-1 map = 99%, reduce = 19%, Cumulative CPU 735.48 sec
2013-09-17 23:22:36,100 Stage-1 map = 100%, reduce = 28%, Cumulative CPU 749.67 sec
2013-09-17 23:22:37,107 Stage-1 map = 100%, reduce = 30%, Cumulative CPU 749.67 sec
2013-09-17 23:22:38,113 Stage-1 map = 100%, reduce = 31%, Cumulative CPU 749.67 sec
2013-09-17 23:22:39,119 Stage-1 map = 100%, reduce = 31%, Cumulative CPU 749.67 sec
2013-09-17 23:22:40,125 Stage-1 map = 100%, reduce = 31%, Cumulative CPU 749.67 sec
2013-09-17 23:22:41,132 Stage-1 map = 100%, reduce = 49%, Cumulative CPU 749.67 sec
2013-09-17 23:22:42,139 Stage-1 map = 100%, reduce = 49%, Cumulative CPU 749.67 sec
2013-09-17 23:22:43,147 Stage-1 map = 100%, reduce = 52%, Cumulative CPU 759.31 sec
2013-09-17 23:22:44,153 Stage-1 map = 100%, reduce = 62%, Cumulative CPU 792.97 sec
2013-09-17 23:22:45,159 Stage-1 map = 100%, reduce = 69%, Cumulative CPU 792.97 sec
2013-09-17 23:22:46,166 Stage-1 map = 100%, reduce = 80%, Cumulative CPU 805.88 sec
2013-09-17 23:22:47,172 Stage-1 map = 100%, reduce = 82%, Cumulative CPU 816.79 sec
2013-09-17 23:22:48,178 Stage-1 map = 100%, reduce = 88%, Cumulative CPU 816.79 sec
2013-09-17 23:22:49,184 Stage-1 map = 100%, reduce = 93%, Cumulative CPU 829.11 sec
2013-09-17 23:22:50,190 Stage-1 map = 100%, reduce = 95%, Cumulative CPU 843.07 sec
2013-09-17 23:22:51,196 Stage-1 map = 100%, reduce = 98%, Cumulative CPU 872.26 sec
2013-09-17 23:22:52,202 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 872.26 sec
2013-09-17 23:22:53,207 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 872.26 sec
2013-09-17 23:22:54,219 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 872.26 sec
2013-09-17 23:22:55,224 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 872.26 sec
2013-09-17 23:22:56,230 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 897.41 sec
2013-09-17 23:22:57,236 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 897.41 sec
2013-09-17 23:22:58,242 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 897.41 sec
2013-09-17 23:22:59,248 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 897.41 sec
MapReduce Total cumulative CPU time: 14 minutes 57 seconds 410 msec
Ended Job = job_201309172235_0031
Launching Job 2 out of 2
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0032
Hadoop job information for Stage-2: number of mappers: 1; number of reducers: 1
2013-09-17 23:23:01,757 Stage-2 map = 0%, reduce = 0%
2013-09-17 23:23:04,768 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 1.62 sec
2013-09-17 23:23:05,774 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 1.62 sec
2013-09-17 23:23:06,779 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 1.62 sec
2013-09-17 23:23:07,783 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 1.62 sec
2013-09-17 23:23:08,787 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 1.62 sec
2013-09-17 23:23:09,792 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 1.62 sec
2013-09-17 23:23:10,796 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 1.62 sec
2013-09-17 23:23:11,801 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 1.62 sec
2013-09-17 23:23:12,806 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 3.6 sec
2013-09-17 23:23:13,810 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 3.6 sec
MapReduce Total cumulative CPU time: 3 seconds 600 msec
Ended Job = job_201309172235_0032
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 11 Cumulative CPU: 897.41 sec HDFS Read: 773347596 HDFS Write: 200700 SUCCESS
Job 1: Map: 1 Reduce: 1 Cumulative CPU: 3.6 sec HDFS Read: 204277 HDFS Write: 106 SUCCESS
Total MapReduce CPU Time Spent: 15 minutes 1 seconds 10 msec
OK
Time taken: 87.479 seconds, Fetched: 10 row(s)
hive> quit;
times: 2
query: SELECT RegionID, count(DISTINCT UserID) AS u FROM hits_100m GROUP BY RegionID ORDER BY u DESC LIMIT 10;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_27218@mturlrep13_201309172323_828143909.txt
hive> SELECT RegionID, count(DISTINCT UserID) AS u FROM hits_100m GROUP BY RegionID ORDER BY u DESC LIMIT 10;;
Total MapReduce jobs = 2
Launching Job 1 out of 2
Number of reduce tasks not specified. Estimated from input data size: 11
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0033
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 11
2013-09-17 23:23:24,453 Stage-1 map = 0%, reduce = 0%
2013-09-17 23:23:35,535 Stage-1 map = 1%, reduce = 0%
2013-09-17 23:23:36,541 Stage-1 map = 11%, reduce = 0%
2013-09-17 23:23:37,547 Stage-1 map = 15%, reduce = 0%
2013-09-17 23:23:38,823 Stage-1 map = 16%, reduce = 0%
2013-09-17 23:23:39,831 Stage-1 map = 44%, reduce = 0%
2013-09-17 23:23:40,851 Stage-1 map = 52%, reduce = 0%
2013-09-17 23:23:41,857 Stage-1 map = 53%, reduce = 0%
2013-09-17 23:23:42,879 Stage-1 map = 63%, reduce = 0%, Cumulative CPU 11.88 sec
2013-09-17 23:23:43,887 Stage-1 map = 71%, reduce = 0%, Cumulative CPU 11.88 sec
2013-09-17 23:23:44,916 Stage-1 map = 73%, reduce = 0%, Cumulative CPU 40.11 sec
2013-09-17 23:23:45,931 Stage-1 map = 76%, reduce = 0%, Cumulative CPU 54.11 sec
2013-09-17 23:23:46,948 Stage-1 map = 78%, reduce = 0%, Cumulative CPU 83.97 sec
2013-09-17 23:23:47,955 Stage-1 map = 79%, reduce = 0%, Cumulative CPU 116.76 sec
2013-09-17 23:23:48,966 Stage-1 map = 84%, reduce = 0%, Cumulative CPU 237.46 sec
2013-09-17 23:23:49,982 Stage-1 map = 85%, reduce = 0%, Cumulative CPU 292.37 sec
2013-09-17 23:23:50,990 Stage-1 map = 86%, reduce = 0%, Cumulative CPU 331.01 sec
2013-09-17 23:23:51,996 Stage-1 map = 92%, reduce = 0%, Cumulative CPU 474.07 sec
2013-09-17 23:23:53,002 Stage-1 map = 93%, reduce = 0%, Cumulative CPU 516.53 sec
2013-09-17 23:23:54,009 Stage-1 map = 93%, reduce = 0%, Cumulative CPU 516.53 sec
2013-09-17 23:23:55,018 Stage-1 map = 97%, reduce = 2%, Cumulative CPU 516.53 sec
2013-09-17 23:23:56,024 Stage-1 map = 97%, reduce = 10%, Cumulative CPU 516.53 sec
2013-09-17 23:23:57,030 Stage-1 map = 98%, reduce = 22%, Cumulative CPU 599.75 sec
2013-09-17 23:23:58,037 Stage-1 map = 99%, reduce = 24%, Cumulative CPU 656.33 sec
2013-09-17 23:23:59,043 Stage-1 map = 99%, reduce = 26%, Cumulative CPU 717.14 sec
2013-09-17 23:24:00,049 Stage-1 map = 100%, reduce = 27%, Cumulative CPU 746.73 sec
2013-09-17 23:24:01,056 Stage-1 map = 100%, reduce = 27%, Cumulative CPU 746.73 sec
2013-09-17 23:24:02,063 Stage-1 map = 100%, reduce = 29%, Cumulative CPU 746.73 sec
2013-09-17 23:24:03,069 Stage-1 map = 100%, reduce = 29%, Cumulative CPU 746.73 sec
2013-09-17 23:24:04,075 Stage-1 map = 100%, reduce = 29%, Cumulative CPU 746.73 sec
2013-09-17 23:24:05,081 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 746.73 sec
2013-09-17 23:24:06,087 Stage-1 map = 100%, reduce = 53%, Cumulative CPU 746.73 sec
2013-09-17 23:24:07,094 Stage-1 map = 100%, reduce = 53%, Cumulative CPU 746.73 sec
2013-09-17 23:24:08,102 Stage-1 map = 100%, reduce = 59%, Cumulative CPU 766.33 sec
2013-09-17 23:24:09,109 Stage-1 map = 100%, reduce = 70%, Cumulative CPU 790.42 sec
2013-09-17 23:24:10,115 Stage-1 map = 100%, reduce = 70%, Cumulative CPU 790.42 sec
2013-09-17 23:24:11,122 Stage-1 map = 100%, reduce = 86%, Cumulative CPU 817.26 sec
2013-09-17 23:24:12,128 Stage-1 map = 100%, reduce = 87%, Cumulative CPU 817.26 sec
2013-09-17 23:24:13,134 Stage-1 map = 100%, reduce = 90%, Cumulative CPU 828.67 sec
2013-09-17 23:24:14,141 Stage-1 map = 100%, reduce = 95%, Cumulative CPU 840.81 sec
2013-09-17 23:24:15,147 Stage-1 map = 100%, reduce = 98%, Cumulative CPU 869.4 sec
2013-09-17 23:24:16,152 Stage-1 map = 100%, reduce = 98%, Cumulative CPU 869.4 sec
2013-09-17 23:24:17,158 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 869.4 sec
2013-09-17 23:24:18,592 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 869.4 sec
2013-09-17 23:24:19,724 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 869.4 sec
2013-09-17 23:24:20,729 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 869.4 sec
2013-09-17 23:24:21,735 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 894.47 sec
2013-09-17 23:24:22,742 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 894.47 sec
MapReduce Total cumulative CPU time: 14 minutes 54 seconds 470 msec
Ended Job = job_201309172235_0033
Launching Job 2 out of 2
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0034
Hadoop job information for Stage-2: number of mappers: 1; number of reducers: 1
2013-09-17 23:24:28,624 Stage-2 map = 0%, reduce = 0%
2013-09-17 23:24:30,631 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 1.65 sec
2013-09-17 23:24:31,636 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 1.65 sec
2013-09-17 23:24:32,641 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 1.65 sec
2013-09-17 23:24:33,646 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 1.65 sec
2013-09-17 23:24:34,650 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 1.65 sec
2013-09-17 23:24:35,655 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 1.65 sec
2013-09-17 23:24:36,660 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 1.65 sec
2013-09-17 23:24:37,666 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 1.65 sec
2013-09-17 23:24:38,671 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 3.47 sec
2013-09-17 23:24:39,676 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 3.47 sec
2013-09-17 23:24:40,682 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 3.47 sec
MapReduce Total cumulative CPU time: 3 seconds 470 msec
Ended Job = job_201309172235_0034
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 11 Cumulative CPU: 894.47 sec HDFS Read: 773347596 HDFS Write: 200700 SUCCESS
Job 1: Map: 1 Reduce: 1 Cumulative CPU: 3.47 sec HDFS Read: 204277 HDFS Write: 106 SUCCESS
Total MapReduce CPU Time Spent: 14 minutes 57 seconds 940 msec
OK
Time taken: 85.062 seconds, Fetched: 10 row(s)
hive> quit;
times: 3
query: SELECT RegionID, count(DISTINCT UserID) AS u FROM hits_100m GROUP BY RegionID ORDER BY u DESC LIMIT 10;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_1610@mturlrep13_201309172324_625629875.txt
hive> SELECT RegionID, count(DISTINCT UserID) AS u FROM hits_100m GROUP BY RegionID ORDER BY u DESC LIMIT 10;;
Total MapReduce jobs = 2
Launching Job 1 out of 2
Number of reduce tasks not specified. Estimated from input data size: 11
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0035
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 11
2013-09-17 23:24:50,386 Stage-1 map = 0%, reduce = 0%
2013-09-17 23:25:02,507 Stage-1 map = 1%, reduce = 0%
2013-09-17 23:25:03,535 Stage-1 map = 8%, reduce = 0%
2013-09-17 23:25:04,541 Stage-1 map = 14%, reduce = 0%
2013-09-17 23:25:05,547 Stage-1 map = 22%, reduce = 0%
2013-09-17 23:25:06,553 Stage-1 map = 39%, reduce = 0%
2013-09-17 23:25:07,559 Stage-1 map = 47%, reduce = 0%
2013-09-17 23:25:08,571 Stage-1 map = 56%, reduce = 0%
2013-09-17 23:25:09,583 Stage-1 map = 61%, reduce = 0%, Cumulative CPU 12.22 sec
2013-09-17 23:25:10,598 Stage-1 map = 67%, reduce = 0%, Cumulative CPU 12.22 sec
2013-09-17 23:25:11,612 Stage-1 map = 71%, reduce = 0%, Cumulative CPU 12.22 sec
2013-09-17 23:25:12,618 Stage-1 map = 75%, reduce = 0%, Cumulative CPU 55.11 sec
2013-09-17 23:25:13,632 Stage-1 map = 76%, reduce = 0%, Cumulative CPU 55.11 sec
2013-09-17 23:25:14,646 Stage-1 map = 78%, reduce = 0%, Cumulative CPU 117.12 sec
2013-09-17 23:25:15,672 Stage-1 map = 80%, reduce = 0%, Cumulative CPU 186.45 sec
2013-09-17 23:25:16,681 Stage-1 map = 84%, reduce = 0%, Cumulative CPU 275.56 sec
2013-09-17 23:25:17,690 Stage-1 map = 87%, reduce = 0%, Cumulative CPU 373.55 sec
2013-09-17 23:25:18,698 Stage-1 map = 88%, reduce = 0%, Cumulative CPU 434.47 sec
2013-09-17 23:25:19,704 Stage-1 map = 92%, reduce = 0%, Cumulative CPU 497.78 sec
2013-09-17 23:25:20,710 Stage-1 map = 93%, reduce = 0%, Cumulative CPU 518.9 sec
2013-09-17 23:25:21,717 Stage-1 map = 94%, reduce = 0%, Cumulative CPU 518.9 sec
2013-09-17 23:25:22,723 Stage-1 map = 97%, reduce = 0%, Cumulative CPU 518.9 sec
2013-09-17 23:25:23,729 Stage-1 map = 97%, reduce = 15%, Cumulative CPU 722.2 sec
2013-09-17 23:25:24,736 Stage-1 map = 98%, reduce = 20%, Cumulative CPU 729.94 sec
2013-09-17 23:25:25,742 Stage-1 map = 99%, reduce = 24%, Cumulative CPU 745.51 sec
2013-09-17 23:25:26,748 Stage-1 map = 100%, reduce = 26%, Cumulative CPU 754.8 sec
2013-09-17 23:25:27,760 Stage-1 map = 100%, reduce = 26%, Cumulative CPU 754.8 sec
2013-09-17 23:25:28,769 Stage-1 map = 100%, reduce = 27%, Cumulative CPU 754.8 sec
2013-09-17 23:25:29,776 Stage-1 map = 100%, reduce = 29%, Cumulative CPU 754.8 sec
2013-09-17 23:25:30,782 Stage-1 map = 100%, reduce = 29%, Cumulative CPU 754.8 sec
2013-09-17 23:25:31,787 Stage-1 map = 100%, reduce = 29%, Cumulative CPU 754.8 sec
2013-09-17 23:25:32,794 Stage-1 map = 100%, reduce = 48%, Cumulative CPU 754.8 sec
2013-09-17 23:25:33,800 Stage-1 map = 100%, reduce = 56%, Cumulative CPU 754.8 sec
2013-09-17 23:25:34,806 Stage-1 map = 100%, reduce = 60%, Cumulative CPU 754.8 sec
2013-09-17 23:25:35,814 Stage-1 map = 100%, reduce = 76%, Cumulative CPU 775.22 sec
2013-09-17 23:25:36,820 Stage-1 map = 100%, reduce = 79%, Cumulative CPU 800.08 sec
2013-09-17 23:25:37,826 Stage-1 map = 100%, reduce = 86%, Cumulative CPU 813.13 sec
2013-09-17 23:25:38,833 Stage-1 map = 100%, reduce = 94%, Cumulative CPU 854.96 sec
2013-09-17 23:25:39,839 Stage-1 map = 100%, reduce = 97%, Cumulative CPU 881.01 sec
2013-09-17 23:25:40,845 Stage-1 map = 100%, reduce = 97%, Cumulative CPU 881.01 sec
2013-09-17 23:25:41,850 Stage-1 map = 100%, reduce = 98%, Cumulative CPU 881.01 sec
2013-09-17 23:25:42,856 Stage-1 map = 100%, reduce = 98%, Cumulative CPU 881.01 sec
2013-09-17 23:25:43,862 Stage-1 map = 100%, reduce = 98%, Cumulative CPU 881.01 sec
2013-09-17 23:25:45,177 Stage-1 map = 100%, reduce = 98%, Cumulative CPU 881.01 sec
2013-09-17 23:25:46,251 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 881.01 sec
2013-09-17 23:25:47,256 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 881.01 sec
2013-09-17 23:25:48,262 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 881.01 sec
2013-09-17 23:25:49,268 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 905.59 sec
2013-09-17 23:25:50,273 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 905.59 sec
MapReduce Total cumulative CPU time: 15 minutes 5 seconds 590 msec
Ended Job = job_201309172235_0035
Launching Job 2 out of 2
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0036
Hadoop job information for Stage-2: number of mappers: 1; number of reducers: 1
2013-09-17 23:25:55,561 Stage-2 map = 0%, reduce = 0%
2013-09-17 23:25:57,569 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 1.62 sec
2013-09-17 23:25:58,574 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 1.62 sec
2013-09-17 23:25:59,579 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 1.62 sec
2013-09-17 23:26:00,584 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 1.62 sec
2013-09-17 23:26:01,588 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 1.62 sec
2013-09-17 23:26:02,592 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 1.62 sec
2013-09-17 23:26:03,597 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 1.62 sec
2013-09-17 23:26:04,603 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 1.62 sec
2013-09-17 23:26:05,609 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 3.52 sec
2013-09-17 23:26:06,614 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 3.52 sec
2013-09-17 23:26:07,619 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 3.52 sec
MapReduce Total cumulative CPU time: 3 seconds 520 msec
Ended Job = job_201309172235_0036
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 11 Cumulative CPU: 905.59 sec HDFS Read: 773347596 HDFS Write: 200700 SUCCESS
Job 1: Map: 1 Reduce: 1 Cumulative CPU: 3.52 sec HDFS Read: 204266 HDFS Write: 106 SUCCESS
Total MapReduce CPU Time Spent: 15 minutes 9 seconds 110 msec
OK
Time taken: 85.039 seconds, Fetched: 10 row(s)
hive> quit;
-- агрегация, среднее количество ключей.;
times: 1
query: SELECT RegionID, sum(AdvEngineID), count(*) AS c, avg(ResolutionWidth), count(DISTINCT UserID) FROM hits_100m GROUP BY RegionID ORDER BY c DESC LIMIT 10;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_9511@mturlrep13_201309172326_2040723350.txt
hive> SELECT RegionID, sum(AdvEngineID), count(*) AS c, avg(ResolutionWidth), count(DISTINCT UserID) FROM hits_100m GROUP BY RegionID ORDER BY c DESC LIMIT 10;;
Total MapReduce jobs = 2
Launching Job 1 out of 2
Number of reduce tasks not specified. Estimated from input data size: 11
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0037
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 11
2013-09-17 23:26:25,696 Stage-1 map = 0%, reduce = 0%
2013-09-17 23:26:35,814 Stage-1 map = 3%, reduce = 0%
2013-09-17 23:26:38,841 Stage-1 map = 10%, reduce = 0%
2013-09-17 23:26:39,848 Stage-1 map = 13%, reduce = 0%
2013-09-17 23:26:40,855 Stage-1 map = 16%, reduce = 0%
2013-09-17 23:26:41,883 Stage-1 map = 19%, reduce = 0%
2013-09-17 23:26:42,890 Stage-1 map = 29%, reduce = 0%
2013-09-17 23:26:43,897 Stage-1 map = 32%, reduce = 0%
2013-09-17 23:26:45,223 Stage-1 map = 38%, reduce = 0%
2013-09-17 23:26:46,236 Stage-1 map = 43%, reduce = 0%, Cumulative CPU 14.75 sec
2013-09-17 23:26:47,243 Stage-1 map = 49%, reduce = 0%, Cumulative CPU 14.75 sec
2013-09-17 23:26:48,308 Stage-1 map = 55%, reduce = 0%, Cumulative CPU 14.75 sec
2013-09-17 23:26:49,329 Stage-1 map = 61%, reduce = 0%, Cumulative CPU 14.75 sec
2013-09-17 23:26:50,342 Stage-1 map = 63%, reduce = 0%, Cumulative CPU 14.75 sec
2013-09-17 23:26:51,354 Stage-1 map = 67%, reduce = 0%, Cumulative CPU 14.75 sec
2013-09-17 23:26:52,368 Stage-1 map = 70%, reduce = 0%, Cumulative CPU 14.75 sec
2013-09-17 23:26:53,382 Stage-1 map = 75%, reduce = 0%, Cumulative CPU 74.49 sec
2013-09-17 23:26:54,393 Stage-1 map = 78%, reduce = 0%, Cumulative CPU 113.4 sec
2013-09-17 23:26:55,401 Stage-1 map = 81%, reduce = 0%, Cumulative CPU 135.12 sec
2013-09-17 23:26:56,414 Stage-1 map = 82%, reduce = 0%, Cumulative CPU 176.43 sec
2013-09-17 23:26:57,430 Stage-1 map = 83%, reduce = 0%, Cumulative CPU 194.75 sec
2013-09-17 23:26:58,444 Stage-1 map = 85%, reduce = 0%, Cumulative CPU 238.4 sec
2013-09-17 23:26:59,451 Stage-1 map = 86%, reduce = 0%, Cumulative CPU 277.62 sec
2013-09-17 23:27:00,458 Stage-1 map = 89%, reduce = 0%, Cumulative CPU 418.29 sec
2013-09-17 23:27:01,465 Stage-1 map = 91%, reduce = 0%, Cumulative CPU 447.59 sec
2013-09-17 23:27:02,472 Stage-1 map = 94%, reduce = 0%, Cumulative CPU 578.54 sec
2013-09-17 23:27:03,478 Stage-1 map = 95%, reduce = 0%, Cumulative CPU 657.91 sec
2013-09-17 23:27:04,486 Stage-1 map = 96%, reduce = 2%, Cumulative CPU 657.91 sec
2013-09-17 23:27:05,497 Stage-1 map = 97%, reduce = 12%, Cumulative CPU 726.48 sec
2013-09-17 23:27:06,503 Stage-1 map = 98%, reduce = 12%, Cumulative CPU 760.94 sec
2013-09-17 23:27:07,509 Stage-1 map = 99%, reduce = 17%, Cumulative CPU 867.41 sec
2013-09-17 23:27:08,515 Stage-1 map = 99%, reduce = 24%, Cumulative CPU 867.41 sec
2013-09-17 23:27:09,521 Stage-1 map = 100%, reduce = 27%, Cumulative CPU 941.56 sec
2013-09-17 23:27:10,528 Stage-1 map = 100%, reduce = 28%, Cumulative CPU 941.56 sec
2013-09-17 23:27:11,534 Stage-1 map = 100%, reduce = 30%, Cumulative CPU 941.56 sec
2013-09-17 23:27:12,540 Stage-1 map = 100%, reduce = 30%, Cumulative CPU 941.56 sec
2013-09-17 23:27:13,546 Stage-1 map = 100%, reduce = 30%, Cumulative CPU 941.56 sec
2013-09-17 23:27:14,552 Stage-1 map = 100%, reduce = 31%, Cumulative CPU 941.56 sec
2013-09-17 23:27:15,559 Stage-1 map = 100%, reduce = 32%, Cumulative CPU 941.56 sec
2013-09-17 23:27:16,566 Stage-1 map = 100%, reduce = 32%, Cumulative CPU 941.56 sec
2013-09-17 23:27:17,602 Stage-1 map = 100%, reduce = 32%, Cumulative CPU 941.56 sec
2013-09-17 23:27:18,609 Stage-1 map = 100%, reduce = 32%, Cumulative CPU 941.56 sec
2013-09-17 23:27:19,615 Stage-1 map = 100%, reduce = 39%, Cumulative CPU 941.56 sec
2013-09-17 23:27:20,621 Stage-1 map = 100%, reduce = 63%, Cumulative CPU 941.56 sec
2013-09-17 23:27:21,628 Stage-1 map = 100%, reduce = 63%, Cumulative CPU 941.56 sec
2013-09-17 23:27:22,633 Stage-1 map = 100%, reduce = 69%, Cumulative CPU 941.56 sec
2013-09-17 23:27:23,642 Stage-1 map = 100%, reduce = 79%, Cumulative CPU 1072.24 sec
2013-09-17 23:27:24,649 Stage-1 map = 100%, reduce = 81%, Cumulative CPU 1080.16 sec
2013-09-17 23:27:25,655 Stage-1 map = 100%, reduce = 84%, Cumulative CPU 1080.16 sec
2013-09-17 23:27:26,661 Stage-1 map = 100%, reduce = 89%, Cumulative CPU 1087.79 sec
2013-09-17 23:27:27,668 Stage-1 map = 100%, reduce = 92%, Cumulative CPU 1097.18 sec
2013-09-17 23:27:28,675 Stage-1 map = 100%, reduce = 93%, Cumulative CPU 1097.18 sec
2013-09-17 23:27:29,682 Stage-1 map = 100%, reduce = 95%, Cumulative CPU 1106.96 sec
2013-09-17 23:27:30,688 Stage-1 map = 100%, reduce = 96%, Cumulative CPU 1106.96 sec
2013-09-17 23:27:31,695 Stage-1 map = 100%, reduce = 97%, Cumulative CPU 1106.96 sec
2013-09-17 23:27:32,701 Stage-1 map = 100%, reduce = 98%, Cumulative CPU 1106.96 sec
2013-09-17 23:27:33,923 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1136.44 sec
2013-09-17 23:27:34,929 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1136.44 sec
2013-09-17 23:27:35,935 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1136.44 sec
2013-09-17 23:27:36,940 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1136.44 sec
2013-09-17 23:27:37,946 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1136.44 sec
2013-09-17 23:27:41,680 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1136.44 sec
2013-09-17 23:27:42,685 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1136.44 sec
2013-09-17 23:27:43,690 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1136.44 sec
2013-09-17 23:27:45,033 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 1155.81 sec
2013-09-17 23:27:46,039 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 1155.81 sec
2013-09-17 23:27:47,045 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 1155.81 sec
MapReduce Total cumulative CPU time: 19 minutes 15 seconds 810 msec
Ended Job = job_201309172235_0037
Launching Job 2 out of 2
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0038
Hadoop job information for Stage-2: number of mappers: 1; number of reducers: 1
2013-09-17 23:27:49,557 Stage-2 map = 0%, reduce = 0%
2013-09-17 23:27:52,569 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 1.83 sec
2013-09-17 23:27:53,574 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 1.83 sec
2013-09-17 23:27:54,579 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 1.83 sec
2013-09-17 23:27:55,583 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 1.83 sec
2013-09-17 23:27:56,588 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 1.83 sec
2013-09-17 23:27:57,592 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 1.83 sec
2013-09-17 23:27:58,596 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 1.83 sec
2013-09-17 23:27:59,601 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 1.83 sec
2013-09-17 23:28:00,607 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 3.83 sec
2013-09-17 23:28:01,613 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 3.83 sec
MapReduce Total cumulative CPU time: 3 seconds 830 msec
Ended Job = job_201309172235_0038
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 11 Cumulative CPU: 1155.81 sec HDFS Read: 847155129 HDFS Write: 298189 SUCCESS
Job 1: Map: 1 Reduce: 1 Cumulative CPU: 3.83 sec HDFS Read: 301766 HDFS Write: 437 SUCCESS
Total MapReduce CPU Time Spent: 19 minutes 19 seconds 640 msec
OK
Time taken: 106.577 seconds, Fetched: 10 row(s)
hive> quit;
times: 2
query: SELECT RegionID, sum(AdvEngineID), count(*) AS c, avg(ResolutionWidth), count(DISTINCT UserID) FROM hits_100m GROUP BY RegionID ORDER BY c DESC LIMIT 10;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_16855@mturlrep13_201309172328_909540588.txt
hive> SELECT RegionID, sum(AdvEngineID), count(*) AS c, avg(ResolutionWidth), count(DISTINCT UserID) FROM hits_100m GROUP BY RegionID ORDER BY c DESC LIMIT 10;;
Total MapReduce jobs = 2
Launching Job 1 out of 2
Number of reduce tasks not specified. Estimated from input data size: 11
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0039
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 11
2013-09-17 23:28:12,290 Stage-1 map = 0%, reduce = 0%
2013-09-17 23:28:23,414 Stage-1 map = 1%, reduce = 0%, Cumulative CPU 51.91 sec
2013-09-17 23:28:24,442 Stage-1 map = 6%, reduce = 0%, Cumulative CPU 51.91 sec
2013-09-17 23:28:25,480 Stage-1 map = 8%, reduce = 0%, Cumulative CPU 51.91 sec
2013-09-17 23:28:26,503 Stage-1 map = 10%, reduce = 0%, Cumulative CPU 51.91 sec
2013-09-17 23:28:27,510 Stage-1 map = 16%, reduce = 0%, Cumulative CPU 51.91 sec
2013-09-17 23:28:28,517 Stage-1 map = 28%, reduce = 0%, Cumulative CPU 51.91 sec
2013-09-17 23:28:29,524 Stage-1 map = 30%, reduce = 0%, Cumulative CPU 51.91 sec
2013-09-17 23:28:30,530 Stage-1 map = 36%, reduce = 0%, Cumulative CPU 51.91 sec
2013-09-17 23:28:31,537 Stage-1 map = 52%, reduce = 0%, Cumulative CPU 51.91 sec
2013-09-17 23:28:32,543 Stage-1 map = 53%, reduce = 0%, Cumulative CPU 51.91 sec
2013-09-17 23:28:33,550 Stage-1 map = 57%, reduce = 0%, Cumulative CPU 51.91 sec
2013-09-17 23:28:34,558 Stage-1 map = 72%, reduce = 0%, Cumulative CPU 66.92 sec
2013-09-17 23:28:35,564 Stage-1 map = 73%, reduce = 0%, Cumulative CPU 66.92 sec
2013-09-17 23:28:36,571 Stage-1 map = 75%, reduce = 0%, Cumulative CPU 66.92 sec
2013-09-17 23:28:37,576 Stage-1 map = 77%, reduce = 0%, Cumulative CPU 91.33 sec
2013-09-17 23:28:38,583 Stage-1 map = 78%, reduce = 0%, Cumulative CPU 128.84 sec
2013-09-17 23:28:39,589 Stage-1 map = 79%, reduce = 0%, Cumulative CPU 167.09 sec
2013-09-17 23:28:40,601 Stage-1 map = 81%, reduce = 0%, Cumulative CPU 202.64 sec
2013-09-17 23:28:41,608 Stage-1 map = 83%, reduce = 0%, Cumulative CPU 265.38 sec
2013-09-17 23:28:42,615 Stage-1 map = 85%, reduce = 0%, Cumulative CPU 331.39 sec
2013-09-17 23:28:43,629 Stage-1 map = 87%, reduce = 0%, Cumulative CPU 434.98 sec
2013-09-17 23:28:44,639 Stage-1 map = 88%, reduce = 0%, Cumulative CPU 460.38 sec
2013-09-17 23:28:45,647 Stage-1 map = 91%, reduce = 0%, Cumulative CPU 605.85 sec
2013-09-17 23:28:46,653 Stage-1 map = 94%, reduce = 0%, Cumulative CPU 658.68 sec
2013-09-17 23:28:47,659 Stage-1 map = 95%, reduce = 0%, Cumulative CPU 658.68 sec
2013-09-17 23:28:48,665 Stage-1 map = 95%, reduce = 1%, Cumulative CPU 658.68 sec
2013-09-17 23:28:49,672 Stage-1 map = 97%, reduce = 12%, Cumulative CPU 658.68 sec
2013-09-17 23:28:50,678 Stage-1 map = 97%, reduce = 17%, Cumulative CPU 658.68 sec
2013-09-17 23:28:51,684 Stage-1 map = 98%, reduce = 21%, Cumulative CPU 795.29 sec
2013-09-17 23:28:52,690 Stage-1 map = 100%, reduce = 24%, Cumulative CPU 927.85 sec
2013-09-17 23:28:53,696 Stage-1 map = 100%, reduce = 24%, Cumulative CPU 927.85 sec
2013-09-17 23:28:54,702 Stage-1 map = 100%, reduce = 28%, Cumulative CPU 927.85 sec
2013-09-17 23:28:55,712 Stage-1 map = 100%, reduce = 30%, Cumulative CPU 927.85 sec
2013-09-17 23:28:56,718 Stage-1 map = 100%, reduce = 30%, Cumulative CPU 927.85 sec
2013-09-17 23:28:57,724 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 927.85 sec
2013-09-17 23:28:58,730 Stage-1 map = 100%, reduce = 48%, Cumulative CPU 927.85 sec
2013-09-17 23:28:59,737 Stage-1 map = 100%, reduce = 52%, Cumulative CPU 927.85 sec
2013-09-17 23:29:00,765 Stage-1 map = 100%, reduce = 56%, Cumulative CPU 927.85 sec
2013-09-17 23:29:01,771 Stage-1 map = 100%, reduce = 65%, Cumulative CPU 927.85 sec
2013-09-17 23:29:02,777 Stage-1 map = 100%, reduce = 65%, Cumulative CPU 927.85 sec
2013-09-17 23:29:03,785 Stage-1 map = 100%, reduce = 72%, Cumulative CPU 940.95 sec
2013-09-17 23:29:04,793 Stage-1 map = 100%, reduce = 80%, Cumulative CPU 954.71 sec
2013-09-17 23:29:05,799 Stage-1 map = 100%, reduce = 81%, Cumulative CPU 954.71 sec
2013-09-17 23:29:06,958 Stage-1 map = 100%, reduce = 89%, Cumulative CPU 1004.24 sec
2013-09-17 23:29:08,056 Stage-1 map = 100%, reduce = 90%, Cumulative CPU 1004.24 sec
2013-09-17 23:29:09,063 Stage-1 map = 100%, reduce = 93%, Cumulative CPU 1022.86 sec
2013-09-17 23:29:10,069 Stage-1 map = 100%, reduce = 95%, Cumulative CPU 1079.45 sec
2013-09-17 23:29:11,075 Stage-1 map = 100%, reduce = 96%, Cumulative CPU 1079.45 sec
2013-09-17 23:29:12,081 Stage-1 map = 100%, reduce = 96%, Cumulative CPU 1079.45 sec
2013-09-17 23:29:13,086 Stage-1 map = 100%, reduce = 96%, Cumulative CPU 1079.45 sec
2013-09-17 23:29:14,092 Stage-1 map = 100%, reduce = 96%, Cumulative CPU 1079.45 sec
2013-09-17 23:29:15,098 Stage-1 map = 100%, reduce = 97%, Cumulative CPU 1079.45 sec
2013-09-17 23:29:16,103 Stage-1 map = 100%, reduce = 98%, Cumulative CPU 1102.32 sec
2013-09-17 23:29:17,109 Stage-1 map = 100%, reduce = 98%, Cumulative CPU 1102.32 sec
2013-09-17 23:29:18,115 Stage-1 map = 100%, reduce = 98%, Cumulative CPU 1102.32 sec
2013-09-17 23:29:19,120 Stage-1 map = 100%, reduce = 98%, Cumulative CPU 1102.32 sec
2013-09-17 23:29:20,126 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1102.32 sec
2013-09-17 23:29:21,131 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1102.32 sec
2013-09-17 23:29:22,137 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1102.32 sec
2013-09-17 23:29:23,142 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1102.32 sec
2013-09-17 23:29:24,148 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1135.97 sec
2013-09-17 23:29:25,154 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1135.97 sec
2013-09-17 23:29:26,159 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1135.97 sec
2013-09-17 23:29:28,761 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 1141.12 sec
2013-09-17 23:29:29,766 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 1141.12 sec
2013-09-17 23:29:30,771 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 1141.12 sec
MapReduce Total cumulative CPU time: 19 minutes 1 seconds 120 msec
Ended Job = job_201309172235_0039
Launching Job 2 out of 2
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0040
Hadoop job information for Stage-2: number of mappers: 1; number of reducers: 1
2013-09-17 23:29:33,329 Stage-2 map = 0%, reduce = 0%
2013-09-17 23:29:36,340 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 1.71 sec
2013-09-17 23:29:37,344 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 1.71 sec
2013-09-17 23:29:38,348 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 1.71 sec
2013-09-17 23:29:39,352 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 1.71 sec
2013-09-17 23:29:40,356 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 1.71 sec
2013-09-17 23:29:41,360 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 1.71 sec
2013-09-17 23:29:42,364 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 1.71 sec
2013-09-17 23:29:43,369 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 1.71 sec
2013-09-17 23:29:44,373 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 3.59 sec
2013-09-17 23:29:45,378 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 3.59 sec
2013-09-17 23:29:46,383 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 3.59 sec
MapReduce Total cumulative CPU time: 3 seconds 590 msec
Ended Job = job_201309172235_0040
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 11 Cumulative CPU: 1141.12 sec HDFS Read: 847155129 HDFS Write: 298189 SUCCESS
Job 1: Map: 1 Reduce: 1 Cumulative CPU: 3.59 sec HDFS Read: 301766 HDFS Write: 437 SUCCESS
Total MapReduce CPU Time Spent: 19 minutes 4 seconds 710 msec
OK
Time taken: 102.879 seconds, Fetched: 10 row(s)
hive> quit;
times: 3
query: SELECT RegionID, sum(AdvEngineID), count(*) AS c, avg(ResolutionWidth), count(DISTINCT UserID) FROM hits_100m GROUP BY RegionID ORDER BY c DESC LIMIT 10;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_24207@mturlrep13_201309172329_822092081.txt
hive> SELECT RegionID, sum(AdvEngineID), count(*) AS c, avg(ResolutionWidth), count(DISTINCT UserID) FROM hits_100m GROUP BY RegionID ORDER BY c DESC LIMIT 10;;
Total MapReduce jobs = 2
Launching Job 1 out of 2
Number of reduce tasks not specified. Estimated from input data size: 11
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0041
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 11
2013-09-17 23:29:56,995 Stage-1 map = 0%, reduce = 0%
2013-09-17 23:30:09,088 Stage-1 map = 5%, reduce = 0%
2013-09-17 23:30:10,094 Stage-1 map = 6%, reduce = 0%
2013-09-17 23:30:11,105 Stage-1 map = 7%, reduce = 0%
2013-09-17 23:30:12,113 Stage-1 map = 15%, reduce = 0%
2013-09-17 23:30:13,119 Stage-1 map = 26%, reduce = 0%
2013-09-17 23:30:14,124 Stage-1 map = 29%, reduce = 0%
2013-09-17 23:30:15,133 Stage-1 map = 33%, reduce = 0%
2013-09-17 23:30:16,147 Stage-1 map = 47%, reduce = 0%
2013-09-17 23:30:17,156 Stage-1 map = 52%, reduce = 0%
2013-09-17 23:30:18,165 Stage-1 map = 54%, reduce = 0%
2013-09-17 23:30:19,192 Stage-1 map = 68%, reduce = 0%, Cumulative CPU 14.3 sec
2013-09-17 23:30:20,200 Stage-1 map = 73%, reduce = 0%, Cumulative CPU 14.3 sec
2013-09-17 23:30:21,213 Stage-1 map = 73%, reduce = 0%, Cumulative CPU 14.3 sec
2013-09-17 23:30:22,219 Stage-1 map = 76%, reduce = 0%, Cumulative CPU 32.42 sec
2013-09-17 23:30:23,227 Stage-1 map = 78%, reduce = 0%, Cumulative CPU 70.55 sec
2013-09-17 23:30:24,234 Stage-1 map = 78%, reduce = 0%, Cumulative CPU 668.79 sec
2013-09-17 23:30:25,259 Stage-1 map = 79%, reduce = 0%, Cumulative CPU 673.33 sec
2013-09-17 23:30:26,274 Stage-1 map = 81%, reduce = 0%, Cumulative CPU 678.94 sec
2013-09-17 23:30:27,300 Stage-1 map = 84%, reduce = 0%, Cumulative CPU 702.3 sec
2013-09-17 23:30:28,312 Stage-1 map = 85%, reduce = 0%, Cumulative CPU 712.73 sec
2013-09-17 23:30:29,319 Stage-1 map = 88%, reduce = 0%, Cumulative CPU 738.35 sec
2013-09-17 23:30:30,326 Stage-1 map = 90%, reduce = 0%, Cumulative CPU 769.71 sec
2013-09-17 23:30:31,333 Stage-1 map = 91%, reduce = 0%, Cumulative CPU 778.96 sec
2013-09-17 23:30:32,339 Stage-1 map = 95%, reduce = 0%, Cumulative CPU 797.52 sec
2013-09-17 23:30:33,345 Stage-1 map = 95%, reduce = 1%, Cumulative CPU 797.52 sec
2013-09-17 23:30:34,354 Stage-1 map = 95%, reduce = 8%, Cumulative CPU 797.52 sec
2013-09-17 23:30:35,361 Stage-1 map = 97%, reduce = 13%, Cumulative CPU 797.52 sec
2013-09-17 23:30:36,367 Stage-1 map = 98%, reduce = 15%, Cumulative CPU 841.63 sec
2013-09-17 23:30:37,373 Stage-1 map = 98%, reduce = 21%, Cumulative CPU 857.24 sec
2013-09-17 23:30:38,379 Stage-1 map = 100%, reduce = 27%, Cumulative CPU 928.19 sec
2013-09-17 23:30:39,385 Stage-1 map = 100%, reduce = 27%, Cumulative CPU 928.19 sec
2013-09-17 23:30:40,391 Stage-1 map = 100%, reduce = 29%, Cumulative CPU 928.19 sec
2013-09-17 23:30:41,396 Stage-1 map = 100%, reduce = 29%, Cumulative CPU 928.19 sec
2013-09-17 23:30:42,403 Stage-1 map = 100%, reduce = 29%, Cumulative CPU 928.19 sec
2013-09-17 23:30:43,409 Stage-1 map = 100%, reduce = 30%, Cumulative CPU 928.19 sec
2013-09-17 23:30:44,415 Stage-1 map = 100%, reduce = 37%, Cumulative CPU 928.19 sec
2013-09-17 23:30:45,421 Stage-1 map = 100%, reduce = 37%, Cumulative CPU 928.19 sec
2013-09-17 23:30:46,428 Stage-1 map = 100%, reduce = 37%, Cumulative CPU 928.19 sec
2013-09-17 23:30:47,451 Stage-1 map = 100%, reduce = 44%, Cumulative CPU 928.19 sec
2013-09-17 23:30:48,460 Stage-1 map = 100%, reduce = 48%, Cumulative CPU 940.63 sec
2013-09-17 23:30:49,467 Stage-1 map = 100%, reduce = 66%, Cumulative CPU 940.63 sec
2013-09-17 23:30:50,474 Stage-1 map = 100%, reduce = 71%, Cumulative CPU 940.63 sec
2013-09-17 23:30:51,480 Stage-1 map = 100%, reduce = 72%, Cumulative CPU 940.63 sec
2013-09-17 23:30:52,487 Stage-1 map = 100%, reduce = 81%, Cumulative CPU 940.63 sec
2013-09-17 23:30:53,494 Stage-1 map = 100%, reduce = 85%, Cumulative CPU 954.04 sec
2013-09-17 23:30:54,500 Stage-1 map = 100%, reduce = 86%, Cumulative CPU 989.38 sec
2013-09-17 23:30:55,507 Stage-1 map = 100%, reduce = 91%, Cumulative CPU 989.38 sec
2013-09-17 23:30:56,514 Stage-1 map = 100%, reduce = 93%, Cumulative CPU 1005.72 sec
2013-09-17 23:30:57,520 Stage-1 map = 100%, reduce = 93%, Cumulative CPU 1005.72 sec
2013-09-17 23:30:58,526 Stage-1 map = 100%, reduce = 96%, Cumulative CPU 1040.57 sec
2013-09-17 23:30:59,532 Stage-1 map = 100%, reduce = 97%, Cumulative CPU 1061.53 sec
2013-09-17 23:31:00,538 Stage-1 map = 100%, reduce = 98%, Cumulative CPU 1081.7 sec
2013-09-17 23:31:01,544 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1104.84 sec
2013-09-17 23:31:02,550 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1104.84 sec
2013-09-17 23:31:03,556 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1104.84 sec
2013-09-17 23:31:04,562 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1104.84 sec
2013-09-17 23:31:05,568 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1104.84 sec
2013-09-17 23:31:06,573 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1104.84 sec
2013-09-17 23:31:07,579 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1104.84 sec
2013-09-17 23:31:08,585 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1104.84 sec
2013-09-17 23:31:09,591 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1104.84 sec
2013-09-17 23:31:10,597 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1104.84 sec
2013-09-17 23:31:13,163 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 1144.43 sec
2013-09-17 23:31:14,168 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 1144.43 sec
MapReduce Total cumulative CPU time: 19 minutes 4 seconds 430 msec
Ended Job = job_201309172235_0041
Launching Job 2 out of 2
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0042
Hadoop job information for Stage-2: number of mappers: 1; number of reducers: 1
2013-09-17 23:31:17,701 Stage-2 map = 0%, reduce = 0%
2013-09-17 23:31:19,709 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 1.75 sec
2013-09-17 23:31:20,714 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 1.75 sec
2013-09-17 23:31:21,719 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 1.75 sec
2013-09-17 23:31:22,724 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 1.75 sec
2013-09-17 23:31:23,728 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 1.75 sec
2013-09-17 23:31:24,733 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 1.75 sec
2013-09-17 23:31:25,738 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 1.75 sec
2013-09-17 23:31:26,743 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 1.75 sec
2013-09-17 23:31:27,748 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 3.69 sec
2013-09-17 23:31:28,754 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 3.69 sec
2013-09-17 23:31:29,759 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 3.69 sec
MapReduce Total cumulative CPU time: 3 seconds 690 msec
Ended Job = job_201309172235_0042
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 11 Cumulative CPU: 1144.43 sec HDFS Read: 847155129 HDFS Write: 298189 SUCCESS
Job 1: Map: 1 Reduce: 1 Cumulative CPU: 3.69 sec HDFS Read: 301766 HDFS Write: 437 SUCCESS
Total MapReduce CPU Time Spent: 19 minutes 8 seconds 120 msec
OK
Time taken: 101.705 seconds, Fetched: 10 row(s)
hive> quit;
-- агрегация, среднее количество ключей, несколько агрегатных функций.;
times: 1
query: SELECT MobilePhoneModel, count(DISTINCT UserID) AS u FROM hits_100m WHERE MobilePhoneModel != '' GROUP BY MobilePhoneModel ORDER BY u DESC LIMIT 10;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_32236@mturlrep13_201309172331_1074251986.txt
hive> SELECT MobilePhoneModel, count(DISTINCT UserID) AS u FROM hits_100m WHERE MobilePhoneModel != '' GROUP BY MobilePhoneModel ORDER BY u DESC LIMIT 10;;
Total MapReduce jobs = 2
Launching Job 1 out of 2
Number of reduce tasks not specified. Estimated from input data size: 11
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0043
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 11
2013-09-17 23:31:47,285 Stage-1 map = 0%, reduce = 0%
2013-09-17 23:31:57,397 Stage-1 map = 7%, reduce = 0%
2013-09-17 23:31:58,403 Stage-1 map = 8%, reduce = 0%
2013-09-17 23:32:01,416 Stage-1 map = 23%, reduce = 0%
2013-09-17 23:32:02,422 Stage-1 map = 24%, reduce = 0%
2013-09-17 23:32:03,436 Stage-1 map = 25%, reduce = 0%, Cumulative CPU 7.06 sec
2013-09-17 23:32:04,442 Stage-1 map = 49%, reduce = 0%, Cumulative CPU 7.06 sec
2013-09-17 23:32:06,734 Stage-1 map = 50%, reduce = 0%, Cumulative CPU 17.18 sec
2013-09-17 23:32:07,740 Stage-1 map = 66%, reduce = 0%, Cumulative CPU 57.06 sec
2013-09-17 23:32:08,752 Stage-1 map = 66%, reduce = 0%, Cumulative CPU 57.06 sec
2013-09-17 23:32:09,767 Stage-1 map = 69%, reduce = 0%, Cumulative CPU 67.22 sec
2013-09-17 23:32:10,784 Stage-1 map = 89%, reduce = 0%, Cumulative CPU 135.37 sec
2013-09-17 23:32:11,792 Stage-1 map = 95%, reduce = 0%, Cumulative CPU 293.42 sec
2013-09-17 23:32:12,798 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 369.89 sec
2013-09-17 23:32:13,805 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 369.89 sec
2013-09-17 23:32:14,811 Stage-1 map = 100%, reduce = 6%, Cumulative CPU 369.89 sec
2013-09-17 23:32:15,817 Stage-1 map = 100%, reduce = 6%, Cumulative CPU 369.89 sec
2013-09-17 23:32:16,825 Stage-1 map = 100%, reduce = 18%, Cumulative CPU 374.04 sec
2013-09-17 23:32:17,832 Stage-1 map = 100%, reduce = 30%, Cumulative CPU 378.73 sec
2013-09-17 23:32:18,840 Stage-1 map = 100%, reduce = 49%, Cumulative CPU 387.59 sec
2013-09-17 23:32:19,848 Stage-1 map = 100%, reduce = 73%, Cumulative CPU 396.86 sec
2013-09-17 23:32:20,855 Stage-1 map = 100%, reduce = 94%, Cumulative CPU 418.82 sec
2013-09-17 23:32:21,861 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 425.02 sec
2013-09-17 23:32:22,868 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 425.02 sec
MapReduce Total cumulative CPU time: 7 minutes 5 seconds 20 msec
Ended Job = job_201309172235_0043
Launching Job 2 out of 2
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0044
Hadoop job information for Stage-2: number of mappers: 1; number of reducers: 1
2013-09-17 23:32:26,376 Stage-2 map = 0%, reduce = 0%
2013-09-17 23:32:28,387 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.98 sec
2013-09-17 23:32:29,392 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.98 sec
2013-09-17 23:32:30,398 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.98 sec
2013-09-17 23:32:31,403 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.98 sec
2013-09-17 23:32:32,408 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.98 sec
2013-09-17 23:32:33,413 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.98 sec
2013-09-17 23:32:34,419 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.98 sec
2013-09-17 23:32:35,425 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 0.98 sec
2013-09-17 23:32:36,430 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 2.64 sec
2013-09-17 23:32:37,437 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 2.64 sec
MapReduce Total cumulative CPU time: 2 seconds 640 msec
Ended Job = job_201309172235_0044
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 11 Cumulative CPU: 425.02 sec HDFS Read: 673615476 HDFS Write: 41947 SUCCESS
Job 1: Map: 1 Reduce: 1 Cumulative CPU: 2.64 sec HDFS Read: 45524 HDFS Write: 131 SUCCESS
Total MapReduce CPU Time Spent: 7 minutes 7 seconds 660 msec
OK
Time taken: 60.4 seconds, Fetched: 10 row(s)
hive> quit;
times: 2
query: SELECT MobilePhoneModel, count(DISTINCT UserID) AS u FROM hits_100m WHERE MobilePhoneModel != '' GROUP BY MobilePhoneModel ORDER BY u DESC LIMIT 10;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_7129@mturlrep13_201309172332_718169783.txt
hive> SELECT MobilePhoneModel, count(DISTINCT UserID) AS u FROM hits_100m WHERE MobilePhoneModel != '' GROUP BY MobilePhoneModel ORDER BY u DESC LIMIT 10;;
Total MapReduce jobs = 2
Launching Job 1 out of 2
Number of reduce tasks not specified. Estimated from input data size: 11
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0045
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 11
2013-09-17 23:32:46,858 Stage-1 map = 0%, reduce = 0%
2013-09-17 23:32:58,979 Stage-1 map = 8%, reduce = 0%
2013-09-17 23:32:59,985 Stage-1 map = 18%, reduce = 0%
2013-09-17 23:33:00,999 Stage-1 map = 22%, reduce = 0%
2013-09-17 23:33:02,014 Stage-1 map = 30%, reduce = 0%
2013-09-17 23:33:03,035 Stage-1 map = 78%, reduce = 0%, Cumulative CPU 39.47 sec
2013-09-17 23:33:04,061 Stage-1 map = 84%, reduce = 0%, Cumulative CPU 127.5 sec
2013-09-17 23:33:05,075 Stage-1 map = 98%, reduce = 0%, Cumulative CPU 389.68 sec
2013-09-17 23:33:06,082 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 414.94 sec
2013-09-17 23:33:07,089 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 414.94 sec
2013-09-17 23:33:08,095 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 414.94 sec
2013-09-17 23:33:09,101 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 414.94 sec
2013-09-17 23:33:10,108 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 414.94 sec
2013-09-17 23:33:11,114 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 414.94 sec
2013-09-17 23:33:12,123 Stage-1 map = 100%, reduce = 15%, Cumulative CPU 414.94 sec
2013-09-17 23:33:13,130 Stage-1 map = 100%, reduce = 24%, Cumulative CPU 414.94 sec
2013-09-17 23:33:14,145 Stage-1 map = 100%, reduce = 58%, Cumulative CPU 436.06 sec
2013-09-17 23:33:15,152 Stage-1 map = 100%, reduce = 73%, Cumulative CPU 446.54 sec
2013-09-17 23:33:16,160 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 475.11 sec
2013-09-17 23:33:17,166 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 475.11 sec
2013-09-17 23:33:18,173 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 475.11 sec
MapReduce Total cumulative CPU time: 7 minutes 55 seconds 110 msec
Ended Job = job_201309172235_0045
Launching Job 2 out of 2
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0046
Hadoop job information for Stage-2: number of mappers: 1; number of reducers: 1
2013-09-17 23:33:21,703 Stage-2 map = 0%, reduce = 0%
2013-09-17 23:33:23,714 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 1.0 sec
2013-09-17 23:33:24,719 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 1.0 sec
2013-09-17 23:33:25,724 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 1.0 sec
2013-09-17 23:33:26,729 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 1.0 sec
2013-09-17 23:33:27,733 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 1.0 sec
2013-09-17 23:33:28,738 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 1.0 sec
2013-09-17 23:33:29,743 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 1.0 sec
2013-09-17 23:33:30,749 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 2.71 sec
2013-09-17 23:33:31,755 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 2.71 sec
2013-09-17 23:33:32,761 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 2.71 sec
MapReduce Total cumulative CPU time: 2 seconds 710 msec
Ended Job = job_201309172235_0046
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 11 Cumulative CPU: 475.11 sec HDFS Read: 673615476 HDFS Write: 41947 SUCCESS
Job 1: Map: 1 Reduce: 1 Cumulative CPU: 2.71 sec HDFS Read: 45524 HDFS Write: 131 SUCCESS
Total MapReduce CPU Time Spent: 7 minutes 57 seconds 820 msec
OK
Time taken: 53.498 seconds, Fetched: 10 row(s)
hive> quit;
times: 3
query: SELECT MobilePhoneModel, count(DISTINCT UserID) AS u FROM hits_100m WHERE MobilePhoneModel != '' GROUP BY MobilePhoneModel ORDER BY u DESC LIMIT 10;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_13985@mturlrep13_201309172333_212079891.txt
hive> SELECT MobilePhoneModel, count(DISTINCT UserID) AS u FROM hits_100m WHERE MobilePhoneModel != '' GROUP BY MobilePhoneModel ORDER BY u DESC LIMIT 10;;
Total MapReduce jobs = 2
Launching Job 1 out of 2
Number of reduce tasks not specified. Estimated from input data size: 11
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0047
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 11
2013-09-17 23:33:42,330 Stage-1 map = 0%, reduce = 0%
2013-09-17 23:33:53,413 Stage-1 map = 1%, reduce = 0%
2013-09-17 23:33:54,432 Stage-1 map = 7%, reduce = 0%
2013-09-17 23:33:55,462 Stage-1 map = 15%, reduce = 0%
2013-09-17 23:33:56,469 Stage-1 map = 23%, reduce = 0%
2013-09-17 23:33:57,482 Stage-1 map = 42%, reduce = 0%
2013-09-17 23:33:58,503 Stage-1 map = 69%, reduce = 0%, Cumulative CPU 9.27 sec
2013-09-17 23:33:59,519 Stage-1 map = 83%, reduce = 0%, Cumulative CPU 114.64 sec
2013-09-17 23:34:00,529 Stage-1 map = 98%, reduce = 0%, Cumulative CPU 388.28 sec
2013-09-17 23:34:01,538 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 412.66 sec
2013-09-17 23:34:02,545 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 412.66 sec
2013-09-17 23:34:03,551 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 412.66 sec
2013-09-17 23:34:04,558 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 412.66 sec
2013-09-17 23:34:05,563 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 412.66 sec
2013-09-17 23:34:06,569 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 412.66 sec
2013-09-17 23:34:07,576 Stage-1 map = 100%, reduce = 9%, Cumulative CPU 412.66 sec
2013-09-17 23:34:08,585 Stage-1 map = 100%, reduce = 24%, Cumulative CPU 412.66 sec
2013-09-17 23:34:09,595 Stage-1 map = 100%, reduce = 45%, Cumulative CPU 422.58 sec
2013-09-17 23:34:10,603 Stage-1 map = 100%, reduce = 74%, Cumulative CPU 442.18 sec
2013-09-17 23:34:11,609 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 464.06 sec
2013-09-17 23:34:12,616 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 470.59 sec
2013-09-17 23:34:13,623 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 470.59 sec
MapReduce Total cumulative CPU time: 7 minutes 50 seconds 590 msec
Ended Job = job_201309172235_0047
Launching Job 2 out of 2
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0048
Hadoop job information for Stage-2: number of mappers: 1; number of reducers: 1
2013-09-17 23:34:16,130 Stage-2 map = 0%, reduce = 0%
2013-09-17 23:34:19,144 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.95 sec
2013-09-17 23:34:20,150 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.95 sec
2013-09-17 23:34:21,155 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.95 sec
2013-09-17 23:34:22,160 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.95 sec
2013-09-17 23:34:23,165 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.95 sec
2013-09-17 23:34:24,169 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.95 sec
2013-09-17 23:34:25,174 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.95 sec
2013-09-17 23:34:26,180 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 2.57 sec
2013-09-17 23:34:27,186 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 2.57 sec
2013-09-17 23:34:28,191 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 2.57 sec
MapReduce Total cumulative CPU time: 2 seconds 570 msec
Ended Job = job_201309172235_0048
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 11 Cumulative CPU: 470.59 sec HDFS Read: 673615476 HDFS Write: 41947 SUCCESS
Job 1: Map: 1 Reduce: 1 Cumulative CPU: 2.57 sec HDFS Read: 45524 HDFS Write: 131 SUCCESS
Total MapReduce CPU Time Spent: 7 minutes 53 seconds 160 msec
OK
Time taken: 53.516 seconds, Fetched: 10 row(s)
hive> quit;
-- мощная фильтрация по строкам, затем агрегация по строкам.;
times: 1
query: SELECT MobilePhone, MobilePhoneModel, count(DISTINCT UserID) AS u FROM hits_100m WHERE MobilePhoneModel != '' GROUP BY MobilePhone, MobilePhoneModel ORDER BY u DESC LIMIT 10;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_20877@mturlrep13_201309172334_738248079.txt
hive> SELECT MobilePhone, MobilePhoneModel, count(DISTINCT UserID) AS u FROM hits_100m WHERE MobilePhoneModel != '' GROUP BY MobilePhone, MobilePhoneModel ORDER BY u DESC LIMIT 10;;
Total MapReduce jobs = 2
Launching Job 1 out of 2
Number of reduce tasks not specified. Estimated from input data size: 11
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0049
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 11
2013-09-17 23:34:45,537 Stage-1 map = 0%, reduce = 0%
2013-09-17 23:34:54,639 Stage-1 map = 2%, reduce = 0%
2013-09-17 23:34:55,649 Stage-1 map = 6%, reduce = 0%
2013-09-17 23:34:56,654 Stage-1 map = 9%, reduce = 0%
2013-09-17 23:34:59,668 Stage-1 map = 21%, reduce = 0%
2013-09-17 23:35:01,463 Stage-1 map = 22%, reduce = 0%
2013-09-17 23:35:02,469 Stage-1 map = 43%, reduce = 0%
2013-09-17 23:35:05,662 Stage-1 map = 59%, reduce = 0%, Cumulative CPU 19.08 sec
2013-09-17 23:35:06,670 Stage-1 map = 62%, reduce = 0%, Cumulative CPU 64.52 sec
2013-09-17 23:35:07,678 Stage-1 map = 67%, reduce = 0%, Cumulative CPU 81.56 sec
2013-09-17 23:35:08,689 Stage-1 map = 81%, reduce = 0%, Cumulative CPU 92.37 sec
2013-09-17 23:35:09,697 Stage-1 map = 84%, reduce = 0%, Cumulative CPU 161.79 sec
2013-09-17 23:35:10,703 Stage-1 map = 95%, reduce = 0%, Cumulative CPU 299.59 sec
2013-09-17 23:35:11,709 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 358.46 sec
2013-09-17 23:35:12,716 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 358.46 sec
2013-09-17 23:35:13,722 Stage-1 map = 100%, reduce = 6%, Cumulative CPU 358.46 sec
2013-09-17 23:35:14,729 Stage-1 map = 100%, reduce = 15%, Cumulative CPU 358.46 sec
2013-09-17 23:35:15,735 Stage-1 map = 100%, reduce = 18%, Cumulative CPU 358.46 sec
2013-09-17 23:35:16,745 Stage-1 map = 100%, reduce = 49%, Cumulative CPU 372.22 sec
2013-09-17 23:35:17,752 Stage-1 map = 100%, reduce = 61%, Cumulative CPU 377.09 sec
2013-09-17 23:35:18,760 Stage-1 map = 100%, reduce = 80%, Cumulative CPU 394.56 sec
2013-09-17 23:35:19,766 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 415.94 sec
2013-09-17 23:35:20,773 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 415.94 sec
2013-09-17 23:35:21,779 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 415.94 sec
MapReduce Total cumulative CPU time: 6 minutes 55 seconds 940 msec
Ended Job = job_201309172235_0049
Launching Job 2 out of 2
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0050
Hadoop job information for Stage-2: number of mappers: 1; number of reducers: 1
2013-09-17 23:35:24,308 Stage-2 map = 0%, reduce = 0%
2013-09-17 23:35:27,321 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 1.12 sec
2013-09-17 23:35:28,327 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 1.12 sec
2013-09-17 23:35:29,332 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 1.12 sec
2013-09-17 23:35:30,337 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 1.12 sec
2013-09-17 23:35:31,341 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 1.12 sec
2013-09-17 23:35:32,346 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 1.12 sec
2013-09-17 23:35:33,351 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 1.12 sec
2013-09-17 23:35:34,356 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 1.12 sec
2013-09-17 23:35:35,362 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 2.78 sec
2013-09-17 23:35:36,367 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 2.78 sec
MapReduce Total cumulative CPU time: 2 seconds 780 msec
Ended Job = job_201309172235_0050
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 11 Cumulative CPU: 415.94 sec HDFS Read: 683872957 HDFS Write: 45535 SUCCESS
Job 1: Map: 1 Reduce: 1 Cumulative CPU: 2.78 sec HDFS Read: 49112 HDFS Write: 154 SUCCESS
Total MapReduce CPU Time Spent: 6 minutes 58 seconds 720 msec
OK
Time taken: 61.275 seconds, Fetched: 10 row(s)
hive> quit;
times: 2
query: SELECT MobilePhone, MobilePhoneModel, count(DISTINCT UserID) AS u FROM hits_100m WHERE MobilePhoneModel != '' GROUP BY MobilePhone, MobilePhoneModel ORDER BY u DESC LIMIT 10;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_28396@mturlrep13_201309172335_1701222232.txt
hive> SELECT MobilePhone, MobilePhoneModel, count(DISTINCT UserID) AS u FROM hits_100m WHERE MobilePhoneModel != '' GROUP BY MobilePhone, MobilePhoneModel ORDER BY u DESC LIMIT 10;;
Total MapReduce jobs = 2
Launching Job 1 out of 2
Number of reduce tasks not specified. Estimated from input data size: 11
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0051
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 11
2013-09-17 23:35:47,068 Stage-1 map = 0%, reduce = 0%
2013-09-17 23:35:58,144 Stage-1 map = 4%, reduce = 0%
2013-09-17 23:35:59,176 Stage-1 map = 24%, reduce = 0%
2013-09-17 23:36:00,190 Stage-1 map = 27%, reduce = 0%
2013-09-17 23:36:01,199 Stage-1 map = 35%, reduce = 0%
2013-09-17 23:36:02,206 Stage-1 map = 66%, reduce = 0%
2013-09-17 23:36:03,233 Stage-1 map = 87%, reduce = 0%, Cumulative CPU 156.88 sec
2013-09-17 23:36:04,245 Stage-1 map = 99%, reduce = 0%, Cumulative CPU 396.77 sec
2013-09-17 23:36:05,254 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 408.98 sec
2013-09-17 23:36:06,262 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 408.98 sec
2013-09-17 23:36:07,268 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 408.98 sec
2013-09-17 23:36:08,275 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 408.98 sec
2013-09-17 23:36:09,282 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 408.98 sec
2013-09-17 23:36:10,289 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 408.98 sec
2013-09-17 23:36:11,296 Stage-1 map = 100%, reduce = 15%, Cumulative CPU 408.98 sec
2013-09-17 23:36:12,305 Stage-1 map = 100%, reduce = 24%, Cumulative CPU 408.98 sec
2013-09-17 23:36:13,315 Stage-1 map = 100%, reduce = 52%, Cumulative CPU 424.36 sec
2013-09-17 23:36:14,324 Stage-1 map = 100%, reduce = 67%, Cumulative CPU 435.5 sec
2013-09-17 23:36:15,330 Stage-1 map = 100%, reduce = 98%, Cumulative CPU 463.94 sec
2013-09-17 23:36:16,337 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 471.88 sec
2013-09-17 23:36:17,343 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 471.88 sec
MapReduce Total cumulative CPU time: 7 minutes 51 seconds 880 msec
Ended Job = job_201309172235_0051
Launching Job 2 out of 2
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0052
Hadoop job information for Stage-2: number of mappers: 1; number of reducers: 1
2013-09-17 23:36:20,861 Stage-2 map = 0%, reduce = 0%
2013-09-17 23:36:22,870 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 1.06 sec
2013-09-17 23:36:23,876 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 1.06 sec
2013-09-17 23:36:24,881 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 1.06 sec
2013-09-17 23:36:25,886 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 1.06 sec
2013-09-17 23:36:26,892 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 1.06 sec
2013-09-17 23:36:27,897 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 1.06 sec
2013-09-17 23:36:28,902 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 1.06 sec
2013-09-17 23:36:29,907 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 1.06 sec
2013-09-17 23:36:30,914 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 2.64 sec
2013-09-17 23:36:31,919 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 2.64 sec
MapReduce Total cumulative CPU time: 2 seconds 640 msec
Ended Job = job_201309172235_0052
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 11 Cumulative CPU: 471.88 sec HDFS Read: 683872957 HDFS Write: 45535 SUCCESS
Job 1: Map: 1 Reduce: 1 Cumulative CPU: 2.64 sec HDFS Read: 49112 HDFS Write: 154 SUCCESS
Total MapReduce CPU Time Spent: 7 minutes 54 seconds 520 msec
OK
Time taken: 53.698 seconds, Fetched: 10 row(s)
hive> quit;
times: 3
query: SELECT MobilePhone, MobilePhoneModel, count(DISTINCT UserID) AS u FROM hits_100m WHERE MobilePhoneModel != '' GROUP BY MobilePhone, MobilePhoneModel ORDER BY u DESC LIMIT 10;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_2932@mturlrep13_201309172336_301621711.txt
hive> SELECT MobilePhone, MobilePhoneModel, count(DISTINCT UserID) AS u FROM hits_100m WHERE MobilePhoneModel != '' GROUP BY MobilePhone, MobilePhoneModel ORDER BY u DESC LIMIT 10;;
Total MapReduce jobs = 2
Launching Job 1 out of 2
Number of reduce tasks not specified. Estimated from input data size: 11
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0053
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 11
2013-09-17 23:36:42,472 Stage-1 map = 0%, reduce = 0%
2013-09-17 23:36:53,559 Stage-1 map = 3%, reduce = 0%
2013-09-17 23:36:54,579 Stage-1 map = 23%, reduce = 0%
2013-09-17 23:36:55,590 Stage-1 map = 26%, reduce = 0%
2013-09-17 23:36:56,627 Stage-1 map = 32%, reduce = 0%
2013-09-17 23:36:57,654 Stage-1 map = 80%, reduce = 0%, Cumulative CPU 9.3 sec
2013-09-17 23:36:58,675 Stage-1 map = 86%, reduce = 0%, Cumulative CPU 127.12 sec
2013-09-17 23:36:59,688 Stage-1 map = 98%, reduce = 0%, Cumulative CPU 387.4 sec
2013-09-17 23:37:00,695 Stage-1 map = 99%, reduce = 0%, Cumulative CPU 400.81 sec
2013-09-17 23:37:01,701 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 413.22 sec
2013-09-17 23:37:02,706 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 413.22 sec
2013-09-17 23:37:03,712 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 413.22 sec
2013-09-17 23:37:04,717 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 413.22 sec
2013-09-17 23:37:05,723 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 413.22 sec
2013-09-17 23:37:06,729 Stage-1 map = 100%, reduce = 9%, Cumulative CPU 413.22 sec
2013-09-17 23:37:07,737 Stage-1 map = 100%, reduce = 21%, Cumulative CPU 413.22 sec
2013-09-17 23:37:08,746 Stage-1 map = 100%, reduce = 42%, Cumulative CPU 422.72 sec
2013-09-17 23:37:09,754 Stage-1 map = 100%, reduce = 61%, Cumulative CPU 432.93 sec
2013-09-17 23:37:10,760 Stage-1 map = 100%, reduce = 85%, Cumulative CPU 448.36 sec
2013-09-17 23:37:11,768 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 474.11 sec
2013-09-17 23:37:12,775 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 474.11 sec
MapReduce Total cumulative CPU time: 7 minutes 54 seconds 110 msec
Ended Job = job_201309172235_0053
Launching Job 2 out of 2
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0054
Hadoop job information for Stage-2: number of mappers: 1; number of reducers: 1
2013-09-17 23:37:15,239 Stage-2 map = 0%, reduce = 0%
2013-09-17 23:37:18,252 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 1.03 sec
2013-09-17 23:37:19,258 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 1.03 sec
2013-09-17 23:37:20,263 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 1.03 sec
2013-09-17 23:37:21,269 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 1.03 sec
2013-09-17 23:37:22,274 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 1.03 sec
2013-09-17 23:37:23,279 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 1.03 sec
2013-09-17 23:37:24,284 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 1.03 sec
2013-09-17 23:37:25,289 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 2.5 sec
2013-09-17 23:37:26,295 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 2.5 sec
2013-09-17 23:37:27,301 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 2.5 sec
MapReduce Total cumulative CPU time: 2 seconds 500 msec
Ended Job = job_201309172235_0054
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 11 Cumulative CPU: 474.11 sec HDFS Read: 683872957 HDFS Write: 45535 SUCCESS
Job 1: Map: 1 Reduce: 1 Cumulative CPU: 2.5 sec HDFS Read: 49101 HDFS Write: 154 SUCCESS
Total MapReduce CPU Time Spent: 7 minutes 56 seconds 610 msec
OK
Time taken: 53.577 seconds, Fetched: 10 row(s)
hive> quit;
-- мощная фильтрация по строкам, затем агрегация по паре из числа и строки.;
times: 1
query: SELECT SearchPhrase, count(*) AS c FROM hits_100m WHERE SearchPhrase != '' GROUP BY SearchPhrase ORDER BY c DESC LIMIT 10;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_10187@mturlrep13_201309172337_1774589050.txt
hive> SELECT SearchPhrase, count(*) AS c FROM hits_100m WHERE SearchPhrase != '' GROUP BY SearchPhrase ORDER BY c DESC LIMIT 10;;
Total MapReduce jobs = 2
Launching Job 1 out of 2
Number of reduce tasks not specified. Estimated from input data size: 11
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0055
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 11
2013-09-17 23:37:44,238 Stage-1 map = 0%, reduce = 0%
2013-09-17 23:37:54,302 Stage-1 map = 1%, reduce = 0%
2013-09-17 23:37:55,311 Stage-1 map = 2%, reduce = 0%
2013-09-17 23:37:56,318 Stage-1 map = 5%, reduce = 0%
2013-09-17 23:37:57,890 Stage-1 map = 19%, reduce = 0%
2013-09-17 23:37:58,905 Stage-1 map = 21%, reduce = 0%
2013-09-17 23:37:59,930 Stage-1 map = 25%, reduce = 0%
2013-09-17 23:38:00,937 Stage-1 map = 47%, reduce = 0%
2013-09-17 23:38:01,955 Stage-1 map = 50%, reduce = 0%
2013-09-17 23:38:02,982 Stage-1 map = 56%, reduce = 0%, Cumulative CPU 30.99 sec
2013-09-17 23:38:04,003 Stage-1 map = 77%, reduce = 0%, Cumulative CPU 125.27 sec
2013-09-17 23:38:05,023 Stage-1 map = 81%, reduce = 0%, Cumulative CPU 170.25 sec
2013-09-17 23:38:06,039 Stage-1 map = 87%, reduce = 0%, Cumulative CPU 236.5 sec
2013-09-17 23:38:07,052 Stage-1 map = 95%, reduce = 0%, Cumulative CPU 320.67 sec
2013-09-17 23:38:08,061 Stage-1 map = 99%, reduce = 0%, Cumulative CPU 455.12 sec
2013-09-17 23:38:09,067 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 469.66 sec
2013-09-17 23:38:10,073 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 469.66 sec
2013-09-17 23:38:11,079 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 469.66 sec
2013-09-17 23:38:12,085 Stage-1 map = 100%, reduce = 1%, Cumulative CPU 469.66 sec
2013-09-17 23:38:13,094 Stage-1 map = 100%, reduce = 10%, Cumulative CPU 469.66 sec
2013-09-17 23:38:14,102 Stage-1 map = 100%, reduce = 28%, Cumulative CPU 469.66 sec
2013-09-17 23:38:15,109 Stage-1 map = 100%, reduce = 28%, Cumulative CPU 469.66 sec
2013-09-17 23:38:16,116 Stage-1 map = 100%, reduce = 42%, Cumulative CPU 469.66 sec
2013-09-17 23:38:17,126 Stage-1 map = 100%, reduce = 61%, Cumulative CPU 469.66 sec
2013-09-17 23:38:18,133 Stage-1 map = 100%, reduce = 68%, Cumulative CPU 469.66 sec
2013-09-17 23:38:19,141 Stage-1 map = 100%, reduce = 78%, Cumulative CPU 469.66 sec
2013-09-17 23:38:20,149 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 545.11 sec
2013-09-17 23:38:21,156 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 588.58 sec
2013-09-17 23:38:22,163 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 588.58 sec
MapReduce Total cumulative CPU time: 9 minutes 48 seconds 580 msec
Ended Job = job_201309172235_0055
Launching Job 2 out of 2
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0056
Hadoop job information for Stage-2: number of mappers: 2; number of reducers: 1
2013-09-17 23:38:25,617 Stage-2 map = 0%, reduce = 0%
2013-09-17 23:38:32,641 Stage-2 map = 18%, reduce = 0%
2013-09-17 23:38:35,652 Stage-2 map = 37%, reduce = 0%
2013-09-17 23:38:38,663 Stage-2 map = 55%, reduce = 0%
2013-09-17 23:38:41,672 Stage-2 map = 73%, reduce = 0%
2013-09-17 23:38:44,684 Stage-2 map = 92%, reduce = 0%, Cumulative CPU 25.74 sec
2013-09-17 23:38:45,690 Stage-2 map = 92%, reduce = 0%, Cumulative CPU 25.74 sec
2013-09-17 23:38:46,696 Stage-2 map = 92%, reduce = 0%, Cumulative CPU 25.74 sec
2013-09-17 23:38:47,700 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 54.5 sec
2013-09-17 23:38:48,705 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 54.5 sec
2013-09-17 23:38:49,711 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 54.5 sec
2013-09-17 23:38:50,716 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 54.5 sec
2013-09-17 23:38:51,722 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 54.5 sec
2013-09-17 23:38:52,728 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 54.5 sec
2013-09-17 23:38:53,732 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 54.5 sec
2013-09-17 23:38:54,738 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 54.5 sec
2013-09-17 23:38:55,744 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 54.5 sec
2013-09-17 23:38:56,749 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 54.5 sec
2013-09-17 23:38:57,754 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 54.5 sec
2013-09-17 23:38:58,759 Stage-2 map = 100%, reduce = 74%, Cumulative CPU 54.5 sec
2013-09-17 23:38:59,764 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 64.07 sec
2013-09-17 23:39:00,769 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 64.07 sec
2013-09-17 23:39:01,774 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 64.07 sec
MapReduce Total cumulative CPU time: 1 minutes 4 seconds 70 msec
Ended Job = job_201309172235_0056
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 11 Cumulative CPU: 588.58 sec HDFS Read: 182281719 HDFS Write: 470238755 SUCCESS
Job 1: Map: 2 Reduce: 1 Cumulative CPU: 64.07 sec HDFS Read: 470242477 HDFS Write: 294 SUCCESS
Total MapReduce CPU Time Spent: 10 minutes 52 seconds 650 msec
OK
Time taken: 87.924 seconds, Fetched: 10 row(s)
hive> quit;
times: 2
query: SELECT SearchPhrase, count(*) AS c FROM hits_100m WHERE SearchPhrase != '' GROUP BY SearchPhrase ORDER BY c DESC LIMIT 10;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_16997@mturlrep13_201309172339_969560501.txt
hive> SELECT SearchPhrase, count(*) AS c FROM hits_100m WHERE SearchPhrase != '' GROUP BY SearchPhrase ORDER BY c DESC LIMIT 10;;
Total MapReduce jobs = 2
Launching Job 1 out of 2
Number of reduce tasks not specified. Estimated from input data size: 11
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0057
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 11
2013-09-17 23:39:11,317 Stage-1 map = 0%, reduce = 0%
2013-09-17 23:39:22,427 Stage-1 map = 1%, reduce = 0%
2013-09-17 23:39:23,436 Stage-1 map = 2%, reduce = 0%
2013-09-17 23:39:24,449 Stage-1 map = 17%, reduce = 0%
2013-09-17 23:39:25,489 Stage-1 map = 27%, reduce = 0%, Cumulative CPU 238.77 sec
2013-09-17 23:39:26,502 Stage-1 map = 31%, reduce = 0%, Cumulative CPU 238.77 sec
2013-09-17 23:39:27,523 Stage-1 map = 55%, reduce = 0%, Cumulative CPU 240.99 sec
2013-09-17 23:39:28,534 Stage-1 map = 66%, reduce = 0%, Cumulative CPU 257.1 sec
2013-09-17 23:39:29,549 Stage-1 map = 74%, reduce = 0%, Cumulative CPU 285.48 sec
2013-09-17 23:39:30,573 Stage-1 map = 89%, reduce = 0%, Cumulative CPU 324.5 sec
2013-09-17 23:39:31,585 Stage-1 map = 96%, reduce = 0%, Cumulative CPU 386.01 sec
2013-09-17 23:39:32,592 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 469.29 sec
2013-09-17 23:39:33,597 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 469.29 sec
2013-09-17 23:39:34,603 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 469.29 sec
2013-09-17 23:39:35,608 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 469.29 sec
2013-09-17 23:39:36,614 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 469.29 sec
2013-09-17 23:39:37,621 Stage-1 map = 100%, reduce = 12%, Cumulative CPU 469.29 sec
2013-09-17 23:39:38,629 Stage-1 map = 100%, reduce = 27%, Cumulative CPU 469.29 sec
2013-09-17 23:39:39,637 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 469.29 sec
2013-09-17 23:39:40,644 Stage-1 map = 100%, reduce = 42%, Cumulative CPU 469.29 sec
2013-09-17 23:39:41,652 Stage-1 map = 100%, reduce = 58%, Cumulative CPU 469.29 sec
2013-09-17 23:39:42,659 Stage-1 map = 100%, reduce = 67%, Cumulative CPU 469.29 sec
2013-09-17 23:39:43,669 Stage-1 map = 100%, reduce = 75%, Cumulative CPU 479.78 sec
2013-09-17 23:39:44,677 Stage-1 map = 100%, reduce = 89%, Cumulative CPU 523.18 sec
2013-09-17 23:39:45,684 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 589.52 sec
2013-09-17 23:39:46,692 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 589.52 sec
2013-09-17 23:39:47,699 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 589.52 sec
MapReduce Total cumulative CPU time: 9 minutes 49 seconds 520 msec
Ended Job = job_201309172235_0057
Launching Job 2 out of 2
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0058
Hadoop job information for Stage-2: number of mappers: 2; number of reducers: 1
2013-09-17 23:39:50,186 Stage-2 map = 0%, reduce = 0%
2013-09-17 23:39:58,217 Stage-2 map = 18%, reduce = 0%
2013-09-17 23:40:01,228 Stage-2 map = 37%, reduce = 0%
2013-09-17 23:40:04,240 Stage-2 map = 55%, reduce = 0%
2013-09-17 23:40:07,251 Stage-2 map = 82%, reduce = 0%
2013-09-17 23:40:11,265 Stage-2 map = 92%, reduce = 0%, Cumulative CPU 25.49 sec
2013-09-17 23:40:12,270 Stage-2 map = 92%, reduce = 0%, Cumulative CPU 25.49 sec
2013-09-17 23:40:13,276 Stage-2 map = 92%, reduce = 0%, Cumulative CPU 25.49 sec
2013-09-17 23:40:14,281 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 54.65 sec
2013-09-17 23:40:15,286 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 54.65 sec
2013-09-17 23:40:16,291 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 54.65 sec
2013-09-17 23:40:18,615 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 54.65 sec
2013-09-17 23:40:19,621 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 54.65 sec
2013-09-17 23:40:20,627 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 54.65 sec
2013-09-17 23:40:21,632 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 54.65 sec
2013-09-17 23:40:22,637 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 54.65 sec
2013-09-17 23:40:23,643 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 54.65 sec
2013-09-17 23:40:24,649 Stage-2 map = 100%, reduce = 74%, Cumulative CPU 54.65 sec
2013-09-17 23:40:25,654 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 64.01 sec
2013-09-17 23:40:26,660 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 64.01 sec
MapReduce Total cumulative CPU time: 1 minutes 4 seconds 10 msec
Ended Job = job_201309172235_0058
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 11 Cumulative CPU: 589.52 sec HDFS Read: 182281719 HDFS Write: 470238755 SUCCESS
Job 1: Map: 2 Reduce: 1 Cumulative CPU: 64.01 sec HDFS Read: 470242477 HDFS Write: 294 SUCCESS
Total MapReduce CPU Time Spent: 10 minutes 53 seconds 530 msec
OK
Time taken: 82.999 seconds, Fetched: 10 row(s)
hive> quit;
times: 3
query: SELECT SearchPhrase, count(*) AS c FROM hits_100m WHERE SearchPhrase != '' GROUP BY SearchPhrase ORDER BY c DESC LIMIT 10;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_24505@mturlrep13_201309172340_635324541.txt
hive> SELECT SearchPhrase, count(*) AS c FROM hits_100m WHERE SearchPhrase != '' GROUP BY SearchPhrase ORDER BY c DESC LIMIT 10;;
Total MapReduce jobs = 2
Launching Job 1 out of 2
Number of reduce tasks not specified. Estimated from input data size: 11
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0059
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 11
2013-09-17 23:40:37,367 Stage-1 map = 0%, reduce = 0%
2013-09-17 23:40:48,543 Stage-1 map = 3%, reduce = 0%
2013-09-17 23:40:49,571 Stage-1 map = 11%, reduce = 0%
2013-09-17 23:40:50,591 Stage-1 map = 26%, reduce = 0%
2013-09-17 23:40:51,619 Stage-1 map = 32%, reduce = 0%
2013-09-17 23:40:52,626 Stage-1 map = 43%, reduce = 0%
2013-09-17 23:40:53,651 Stage-1 map = 65%, reduce = 0%, Cumulative CPU 28.49 sec
2013-09-17 23:40:54,672 Stage-1 map = 72%, reduce = 0%, Cumulative CPU 114.43 sec
2013-09-17 23:40:55,681 Stage-1 map = 83%, reduce = 0%, Cumulative CPU 171.5 sec
2013-09-17 23:40:56,691 Stage-1 map = 94%, reduce = 0%, Cumulative CPU 279.83 sec
2013-09-17 23:40:57,706 Stage-1 map = 98%, reduce = 0%, Cumulative CPU 424.96 sec
2013-09-17 23:40:58,712 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 469.69 sec
2013-09-17 23:40:59,717 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 469.69 sec
2013-09-17 23:41:00,723 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 469.69 sec
2013-09-17 23:41:01,728 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 469.69 sec
2013-09-17 23:41:02,735 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 469.69 sec
2013-09-17 23:41:03,743 Stage-1 map = 100%, reduce = 24%, Cumulative CPU 469.69 sec
2013-09-17 23:41:04,751 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 469.69 sec
2013-09-17 23:41:05,758 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 469.69 sec
2013-09-17 23:41:06,765 Stage-1 map = 100%, reduce = 55%, Cumulative CPU 469.69 sec
2013-09-17 23:41:07,772 Stage-1 map = 100%, reduce = 67%, Cumulative CPU 469.69 sec
2013-09-17 23:41:08,779 Stage-1 map = 100%, reduce = 67%, Cumulative CPU 469.69 sec
2013-09-17 23:41:09,788 Stage-1 map = 100%, reduce = 86%, Cumulative CPU 513.51 sec
2013-09-17 23:41:10,794 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 579.41 sec
2013-09-17 23:41:11,800 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 590.63 sec
2013-09-17 23:41:12,807 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 590.63 sec
MapReduce Total cumulative CPU time: 9 minutes 50 seconds 630 msec
Ended Job = job_201309172235_0059
Launching Job 2 out of 2
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0060
Hadoop job information for Stage-2: number of mappers: 2; number of reducers: 1
2013-09-17 23:41:15,266 Stage-2 map = 0%, reduce = 0%
2013-09-17 23:41:23,289 Stage-2 map = 18%, reduce = 0%
2013-09-17 23:41:26,298 Stage-2 map = 37%, reduce = 0%, Cumulative CPU 21.93 sec
2013-09-17 23:41:27,302 Stage-2 map = 37%, reduce = 0%, Cumulative CPU 21.93 sec
2013-09-17 23:41:28,306 Stage-2 map = 37%, reduce = 0%, Cumulative CPU 21.93 sec
2013-09-17 23:41:29,310 Stage-2 map = 55%, reduce = 0%, Cumulative CPU 21.93 sec
2013-09-17 23:41:30,314 Stage-2 map = 55%, reduce = 0%, Cumulative CPU 21.93 sec
2013-09-17 23:41:31,318 Stage-2 map = 55%, reduce = 0%, Cumulative CPU 21.93 sec
2013-09-17 23:41:32,322 Stage-2 map = 73%, reduce = 0%, Cumulative CPU 21.93 sec
2013-09-17 23:41:33,326 Stage-2 map = 73%, reduce = 0%, Cumulative CPU 21.93 sec
2013-09-17 23:41:34,334 Stage-2 map = 73%, reduce = 0%, Cumulative CPU 21.93 sec
2013-09-17 23:41:35,338 Stage-2 map = 92%, reduce = 0%, Cumulative CPU 36.41 sec
2013-09-17 23:41:36,342 Stage-2 map = 92%, reduce = 0%, Cumulative CPU 36.41 sec
2013-09-17 23:41:37,347 Stage-2 map = 92%, reduce = 0%, Cumulative CPU 36.41 sec
2013-09-17 23:41:38,351 Stage-2 map = 92%, reduce = 0%, Cumulative CPU 36.41 sec
2013-09-17 23:41:39,355 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 54.78 sec
2013-09-17 23:41:40,359 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 54.78 sec
2013-09-17 23:41:41,363 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 54.78 sec
2013-09-17 23:41:42,368 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 54.78 sec
2013-09-17 23:41:43,372 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 54.78 sec
2013-09-17 23:41:44,376 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 54.78 sec
2013-09-17 23:41:45,381 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 54.78 sec
2013-09-17 23:41:46,386 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 54.78 sec
2013-09-17 23:41:47,390 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 54.78 sec
2013-09-17 23:41:48,394 Stage-2 map = 100%, reduce = 74%, Cumulative CPU 54.78 sec
2013-09-17 23:41:49,399 Stage-2 map = 100%, reduce = 74%, Cumulative CPU 54.78 sec
2013-09-17 23:41:50,404 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 64.36 sec
2013-09-17 23:41:51,408 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 64.36 sec
MapReduce Total cumulative CPU time: 1 minutes 4 seconds 360 msec
Ended Job = job_201309172235_0060
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 11 Cumulative CPU: 590.63 sec HDFS Read: 182281719 HDFS Write: 470238755 SUCCESS
Job 1: Map: 2 Reduce: 1 Cumulative CPU: 64.36 sec HDFS Read: 470242477 HDFS Write: 294 SUCCESS
Total MapReduce CPU Time Spent: 10 minutes 54 seconds 990 msec
OK
Time taken: 82.867 seconds, Fetched: 10 row(s)
hive> quit;
-- средняя фильтрация по строкам, затем агрегация по строкам, большое количество ключей.;
times: 1
query: SELECT SearchPhrase, count(DISTINCT UserID) AS u FROM hits_100m WHERE SearchPhrase != '' GROUP BY SearchPhrase ORDER BY u DESC LIMIT 10;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_31334@mturlrep13_201309172342_322265508.txt
hive> SELECT SearchPhrase, count(DISTINCT UserID) AS u FROM hits_100m WHERE SearchPhrase != '' GROUP BY SearchPhrase ORDER BY u DESC LIMIT 10;;
Total MapReduce jobs = 2
Launching Job 1 out of 2
Number of reduce tasks not specified. Estimated from input data size: 11
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0061
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 11
2013-09-17 23:42:11,682 Stage-1 map = 0%, reduce = 0%
2013-09-17 23:42:21,798 Stage-1 map = 7%, reduce = 0%
2013-09-17 23:42:22,804 Stage-1 map = 8%, reduce = 0%
2013-09-17 23:42:24,816 Stage-1 map = 15%, reduce = 0%
2013-09-17 23:42:27,083 Stage-1 map = 22%, reduce = 0%, Cumulative CPU 228.9 sec
2013-09-17 23:42:28,092 Stage-1 map = 32%, reduce = 0%, Cumulative CPU 228.9 sec
2013-09-17 23:42:29,101 Stage-1 map = 35%, reduce = 0%, Cumulative CPU 228.9 sec
2013-09-17 23:42:31,162 Stage-1 map = 42%, reduce = 0%, Cumulative CPU 228.9 sec
2013-09-17 23:42:32,201 Stage-1 map = 49%, reduce = 0%, Cumulative CPU 231.63 sec
2013-09-17 23:42:33,214 Stage-1 map = 51%, reduce = 0%, Cumulative CPU 234.97 sec
2013-09-17 23:42:34,223 Stage-1 map = 61%, reduce = 0%, Cumulative CPU 254.73 sec
2013-09-17 23:42:35,315 Stage-1 map = 69%, reduce = 0%, Cumulative CPU 254.73 sec
2013-09-17 23:42:36,327 Stage-1 map = 70%, reduce = 0%, Cumulative CPU 260.85 sec
2013-09-17 23:42:37,339 Stage-1 map = 77%, reduce = 0%, Cumulative CPU 292.13 sec
2013-09-17 23:42:38,347 Stage-1 map = 88%, reduce = 0%, Cumulative CPU 320.32 sec
2013-09-17 23:42:39,358 Stage-1 map = 93%, reduce = 0%, Cumulative CPU 396.48 sec
2013-09-17 23:42:40,364 Stage-1 map = 98%, reduce = 0%, Cumulative CPU 484.46 sec
2013-09-17 23:42:41,369 Stage-1 map = 100%, reduce = 2%, Cumulative CPU 517.16 sec
2013-09-17 23:42:42,376 Stage-1 map = 100%, reduce = 8%, Cumulative CPU 517.16 sec
2013-09-17 23:42:43,383 Stage-1 map = 100%, reduce = 16%, Cumulative CPU 517.16 sec
2013-09-17 23:42:44,391 Stage-1 map = 100%, reduce = 25%, Cumulative CPU 517.16 sec
2013-09-17 23:42:45,397 Stage-1 map = 100%, reduce = 37%, Cumulative CPU 517.16 sec
2013-09-17 23:42:46,404 Stage-1 map = 100%, reduce = 48%, Cumulative CPU 517.16 sec
2013-09-17 23:42:47,411 Stage-1 map = 100%, reduce = 55%, Cumulative CPU 517.16 sec
2013-09-17 23:42:48,419 Stage-1 map = 100%, reduce = 64%, Cumulative CPU 517.16 sec
2013-09-17 23:42:49,427 Stage-1 map = 100%, reduce = 71%, Cumulative CPU 528.88 sec
2013-09-17 23:42:50,435 Stage-1 map = 100%, reduce = 82%, Cumulative CPU 540.66 sec
2013-09-17 23:42:51,442 Stage-1 map = 100%, reduce = 91%, Cumulative CPU 589.36 sec
2013-09-17 23:42:52,449 Stage-1 map = 100%, reduce = 95%, Cumulative CPU 626.57 sec
2013-09-17 23:42:53,455 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 626.57 sec
2013-09-17 23:42:54,461 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 651.04 sec
2013-09-17 23:42:55,468 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 651.04 sec
MapReduce Total cumulative CPU time: 10 minutes 51 seconds 40 msec
Ended Job = job_201309172235_0061
Launching Job 2 out of 2
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0062
Hadoop job information for Stage-2: number of mappers: 2; number of reducers: 1
2013-09-17 23:42:57,983 Stage-2 map = 0%, reduce = 0%
2013-09-17 23:43:06,006 Stage-2 map = 18%, reduce = 0%
2013-09-17 23:43:09,017 Stage-2 map = 37%, reduce = 0%
2013-09-17 23:43:12,028 Stage-2 map = 55%, reduce = 0%
2013-09-17 23:43:15,039 Stage-2 map = 73%, reduce = 0%
2013-09-17 23:43:18,050 Stage-2 map = 82%, reduce = 0%
2013-09-17 23:43:19,055 Stage-2 map = 92%, reduce = 0%, Cumulative CPU 25.26 sec
2013-09-17 23:43:20,061 Stage-2 map = 92%, reduce = 0%, Cumulative CPU 25.26 sec
2013-09-17 23:43:21,068 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 53.98 sec
2013-09-17 23:43:22,073 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 53.98 sec
2013-09-17 23:43:23,077 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 53.98 sec
2013-09-17 23:43:24,082 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 53.98 sec
2013-09-17 23:43:25,087 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 53.98 sec
2013-09-17 23:43:26,092 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 53.98 sec
2013-09-17 23:43:27,096 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 53.98 sec
2013-09-17 23:43:28,101 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 53.98 sec
2013-09-17 23:43:29,106 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 53.98 sec
2013-09-17 23:43:30,110 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 53.98 sec
2013-09-17 23:43:31,114 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 53.98 sec
2013-09-17 23:43:32,119 Stage-2 map = 100%, reduce = 70%, Cumulative CPU 53.98 sec
2013-09-17 23:43:33,124 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 63.69 sec
2013-09-17 23:43:34,129 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 63.69 sec
2013-09-17 23:43:35,135 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 63.69 sec
MapReduce Total cumulative CPU time: 1 minutes 3 seconds 690 msec
Ended Job = job_201309172235_0062
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 11 Cumulative CPU: 651.04 sec HDFS Read: 840872065 HDFS Write: 470236967 SUCCESS
Job 1: Map: 2 Reduce: 1 Cumulative CPU: 63.69 sec HDFS Read: 470240689 HDFS Write: 291 SUCCESS
Total MapReduce CPU Time Spent: 11 minutes 54 seconds 730 msec
OK
Time taken: 94.281 seconds, Fetched: 10 row(s)
hive> quit;
times: 2
query: SELECT SearchPhrase, count(DISTINCT UserID) AS u FROM hits_100m WHERE SearchPhrase != '' GROUP BY SearchPhrase ORDER BY u DESC LIMIT 10;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_6246@mturlrep13_201309172343_752416466.txt
hive> SELECT SearchPhrase, count(DISTINCT UserID) AS u FROM hits_100m WHERE SearchPhrase != '' GROUP BY SearchPhrase ORDER BY u DESC LIMIT 10;;
Total MapReduce jobs = 2
Launching Job 1 out of 2
Number of reduce tasks not specified. Estimated from input data size: 11
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0063
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 11
2013-09-17 23:43:46,269 Stage-1 map = 0%, reduce = 0%
2013-09-17 23:43:55,322 Stage-1 map = 1%, reduce = 0%
2013-09-17 23:43:56,328 Stage-1 map = 2%, reduce = 0%
2013-09-17 23:43:57,335 Stage-1 map = 8%, reduce = 0%
2013-09-17 23:43:58,347 Stage-1 map = 15%, reduce = 0%
2013-09-17 23:43:59,353 Stage-1 map = 20%, reduce = 0%
2013-09-17 23:44:00,368 Stage-1 map = 29%, reduce = 0%
2013-09-17 23:44:01,375 Stage-1 map = 40%, reduce = 0%
2013-09-17 23:44:02,399 Stage-1 map = 46%, reduce = 0%, Cumulative CPU 11.32 sec
2013-09-17 23:44:03,419 Stage-1 map = 54%, reduce = 0%, Cumulative CPU 45.05 sec
2013-09-17 23:44:04,431 Stage-1 map = 73%, reduce = 0%, Cumulative CPU 68.44 sec
2013-09-17 23:44:05,439 Stage-1 map = 78%, reduce = 0%, Cumulative CPU 146.24 sec
2013-09-17 23:44:06,446 Stage-1 map = 86%, reduce = 0%, Cumulative CPU 213.64 sec
2013-09-17 23:44:07,457 Stage-1 map = 93%, reduce = 0%, Cumulative CPU 292.27 sec
2013-09-17 23:44:08,464 Stage-1 map = 98%, reduce = 0%, Cumulative CPU 443.09 sec
2013-09-17 23:44:09,470 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 549.17 sec
2013-09-17 23:44:10,476 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 549.17 sec
2013-09-17 23:44:11,482 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 549.17 sec
2013-09-17 23:44:12,489 Stage-1 map = 100%, reduce = 1%, Cumulative CPU 549.17 sec
2013-09-17 23:44:13,497 Stage-1 map = 100%, reduce = 8%, Cumulative CPU 549.17 sec
2013-09-17 23:44:14,506 Stage-1 map = 100%, reduce = 29%, Cumulative CPU 549.17 sec
2013-09-17 23:44:15,513 Stage-1 map = 100%, reduce = 34%, Cumulative CPU 549.17 sec
2013-09-17 23:44:16,520 Stage-1 map = 100%, reduce = 45%, Cumulative CPU 549.17 sec
2013-09-17 23:44:17,527 Stage-1 map = 100%, reduce = 67%, Cumulative CPU 549.17 sec
2013-09-17 23:44:18,534 Stage-1 map = 100%, reduce = 68%, Cumulative CPU 549.17 sec
2013-09-17 23:44:19,541 Stage-1 map = 100%, reduce = 72%, Cumulative CPU 549.17 sec
2013-09-17 23:44:20,550 Stage-1 map = 100%, reduce = 84%, Cumulative CPU 561.32 sec
2013-09-17 23:44:21,556 Stage-1 map = 100%, reduce = 89%, Cumulative CPU 585.6 sec
2013-09-17 23:44:22,563 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 687.26 sec
2013-09-17 23:44:23,570 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 687.26 sec
2013-09-17 23:44:24,576 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 687.26 sec
MapReduce Total cumulative CPU time: 11 minutes 27 seconds 260 msec
Ended Job = job_201309172235_0063
Launching Job 2 out of 2
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0064
Hadoop job information for Stage-2: number of mappers: 2; number of reducers: 1
2013-09-17 23:44:28,140 Stage-2 map = 0%, reduce = 0%
2013-09-17 23:44:35,163 Stage-2 map = 18%, reduce = 0%
2013-09-17 23:44:38,174 Stage-2 map = 37%, reduce = 0%
2013-09-17 23:44:41,183 Stage-2 map = 55%, reduce = 0%
2013-09-17 23:44:44,194 Stage-2 map = 82%, reduce = 0%
2013-09-17 23:44:47,205 Stage-2 map = 92%, reduce = 0%, Cumulative CPU 25.21 sec
2013-09-17 23:44:48,211 Stage-2 map = 92%, reduce = 0%, Cumulative CPU 25.21 sec
2013-09-17 23:44:49,215 Stage-2 map = 92%, reduce = 0%, Cumulative CPU 25.21 sec
2013-09-17 23:44:50,916 Stage-2 map = 92%, reduce = 0%, Cumulative CPU 25.21 sec
2013-09-17 23:44:51,921 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 54.96 sec
2013-09-17 23:44:52,925 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 54.96 sec
2013-09-17 23:44:53,931 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 54.96 sec
2013-09-17 23:44:54,937 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 54.96 sec
2013-09-17 23:44:55,942 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 54.96 sec
2013-09-17 23:44:56,947 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 54.96 sec
2013-09-17 23:44:57,951 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 54.96 sec
2013-09-17 23:44:58,956 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 54.96 sec
2013-09-17 23:44:59,960 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 54.96 sec
2013-09-17 23:45:00,964 Stage-2 map = 100%, reduce = 70%, Cumulative CPU 54.96 sec
2013-09-17 23:45:01,970 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 64.36 sec
2013-09-17 23:45:02,975 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 64.36 sec
2013-09-17 23:45:03,981 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 64.36 sec
MapReduce Total cumulative CPU time: 1 minutes 4 seconds 360 msec
Ended Job = job_201309172235_0064
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 11 Cumulative CPU: 687.26 sec HDFS Read: 840872065 HDFS Write: 470236967 SUCCESS
Job 1: Map: 2 Reduce: 1 Cumulative CPU: 64.36 sec HDFS Read: 470240689 HDFS Write: 291 SUCCESS
Total MapReduce CPU Time Spent: 12 minutes 31 seconds 620 msec
OK
Time taken: 86.991 seconds, Fetched: 10 row(s)
hive> quit;
times: 3
query: SELECT SearchPhrase, count(DISTINCT UserID) AS u FROM hits_100m WHERE SearchPhrase != '' GROUP BY SearchPhrase ORDER BY u DESC LIMIT 10;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_13649@mturlrep13_201309172345_1690142009.txt
hive> SELECT SearchPhrase, count(DISTINCT UserID) AS u FROM hits_100m WHERE SearchPhrase != '' GROUP BY SearchPhrase ORDER BY u DESC LIMIT 10;;
Total MapReduce jobs = 2
Launching Job 1 out of 2
Number of reduce tasks not specified. Estimated from input data size: 11
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0065
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 11
2013-09-17 23:45:13,718 Stage-1 map = 0%, reduce = 0%
2013-09-17 23:45:24,809 Stage-1 map = 3%, reduce = 0%
2013-09-17 23:45:25,833 Stage-1 map = 9%, reduce = 0%, Cumulative CPU 168.95 sec
2013-09-17 23:45:26,845 Stage-1 map = 12%, reduce = 0%, Cumulative CPU 168.95 sec
2013-09-17 23:45:27,853 Stage-1 map = 23%, reduce = 0%, Cumulative CPU 168.95 sec
2013-09-17 23:45:28,867 Stage-1 map = 29%, reduce = 0%, Cumulative CPU 168.95 sec
2013-09-17 23:45:29,874 Stage-1 map = 40%, reduce = 0%, Cumulative CPU 168.95 sec
2013-09-17 23:45:30,887 Stage-1 map = 44%, reduce = 0%, Cumulative CPU 173.1 sec
2013-09-17 23:45:31,894 Stage-1 map = 56%, reduce = 0%, Cumulative CPU 190.37 sec
2013-09-17 23:45:32,900 Stage-1 map = 69%, reduce = 0%, Cumulative CPU 207.59 sec
2013-09-17 23:45:33,913 Stage-1 map = 75%, reduce = 0%, Cumulative CPU 266.9 sec
2013-09-17 23:45:34,921 Stage-1 map = 83%, reduce = 0%, Cumulative CPU 286.81 sec
2013-09-17 23:45:35,928 Stage-1 map = 88%, reduce = 0%, Cumulative CPU 319.49 sec
2013-09-17 23:45:36,935 Stage-1 map = 93%, reduce = 0%, Cumulative CPU 429.06 sec
2013-09-17 23:45:37,943 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 545.45 sec
2013-09-17 23:45:38,949 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 545.45 sec
2013-09-17 23:45:39,955 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 545.45 sec
2013-09-17 23:45:40,961 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 545.45 sec
2013-09-17 23:45:41,968 Stage-1 map = 100%, reduce = 5%, Cumulative CPU 545.45 sec
2013-09-17 23:45:42,978 Stage-1 map = 100%, reduce = 17%, Cumulative CPU 545.45 sec
2013-09-17 23:45:43,985 Stage-1 map = 100%, reduce = 26%, Cumulative CPU 545.45 sec
2013-09-17 23:45:44,992 Stage-1 map = 100%, reduce = 39%, Cumulative CPU 545.45 sec
2013-09-17 23:45:45,999 Stage-1 map = 100%, reduce = 52%, Cumulative CPU 545.45 sec
2013-09-17 23:45:47,006 Stage-1 map = 100%, reduce = 61%, Cumulative CPU 545.45 sec
2013-09-17 23:45:48,013 Stage-1 map = 100%, reduce = 72%, Cumulative CPU 545.45 sec
2013-09-17 23:45:49,019 Stage-1 map = 100%, reduce = 77%, Cumulative CPU 545.45 sec
2013-09-17 23:45:50,027 Stage-1 map = 100%, reduce = 91%, Cumulative CPU 595.55 sec
2013-09-17 23:45:51,034 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 670.19 sec
2013-09-17 23:45:52,040 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 682.37 sec
2013-09-17 23:45:53,047 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 682.37 sec
MapReduce Total cumulative CPU time: 11 minutes 22 seconds 370 msec
Ended Job = job_201309172235_0065
Launching Job 2 out of 2
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0066
Hadoop job information for Stage-2: number of mappers: 2; number of reducers: 1
2013-09-17 23:45:56,566 Stage-2 map = 0%, reduce = 0%
2013-09-17 23:46:03,591 Stage-2 map = 18%, reduce = 0%
2013-09-17 23:46:06,601 Stage-2 map = 28%, reduce = 0%
2013-09-17 23:46:07,606 Stage-2 map = 37%, reduce = 0%
2013-09-17 23:46:10,617 Stage-2 map = 55%, reduce = 0%
2013-09-17 23:46:13,628 Stage-2 map = 73%, reduce = 0%
2013-09-17 23:46:16,640 Stage-2 map = 92%, reduce = 0%, Cumulative CPU 25.88 sec
2013-09-17 23:46:17,646 Stage-2 map = 92%, reduce = 0%, Cumulative CPU 25.88 sec
2013-09-17 23:46:18,651 Stage-2 map = 92%, reduce = 0%, Cumulative CPU 25.88 sec
2013-09-17 23:46:19,840 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 54.97 sec
2013-09-17 23:46:20,844 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 54.97 sec
2013-09-17 23:46:21,848 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 54.97 sec
2013-09-17 23:46:22,853 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 54.97 sec
2013-09-17 23:46:23,857 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 54.97 sec
2013-09-17 23:46:24,862 Stage-2 map = 100%, reduce = 17%, Cumulative CPU 54.97 sec
2013-09-17 23:46:25,866 Stage-2 map = 100%, reduce = 17%, Cumulative CPU 56.42 sec
2013-09-17 23:46:26,871 Stage-2 map = 100%, reduce = 17%, Cumulative CPU 56.42 sec
2013-09-17 23:46:27,875 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 56.42 sec
2013-09-17 23:46:28,879 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 56.42 sec
2013-09-17 23:46:29,883 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 56.42 sec
2013-09-17 23:46:30,888 Stage-2 map = 100%, reduce = 70%, Cumulative CPU 56.42 sec
2013-09-17 23:46:31,893 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 64.4 sec
2013-09-17 23:46:32,899 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 64.4 sec
MapReduce Total cumulative CPU time: 1 minutes 4 seconds 400 msec
Ended Job = job_201309172235_0066
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 11 Cumulative CPU: 682.37 sec HDFS Read: 840872065 HDFS Write: 470236967 SUCCESS
Job 1: Map: 2 Reduce: 1 Cumulative CPU: 64.4 sec HDFS Read: 470240689 HDFS Write: 291 SUCCESS
Total MapReduce CPU Time Spent: 12 minutes 26 seconds 770 msec
OK
Time taken: 87.084 seconds, Fetched: 10 row(s)
hive> quit;
-- агрегация чуть сложнее.;
times: 1
query: SELECT SearchEngineID, SearchPhrase, count(*) AS c FROM hits_100m WHERE SearchPhrase != '' GROUP BY SearchEngineID, SearchPhrase ORDER BY c DESC LIMIT 10;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_21386@mturlrep13_201309172346_1552848937.txt
hive> SELECT SearchEngineID, SearchPhrase, count(*) AS c FROM hits_100m WHERE SearchPhrase != '' GROUP BY SearchEngineID, SearchPhrase ORDER BY c DESC LIMIT 10;;
Total MapReduce jobs = 2
Launching Job 1 out of 2
Number of reduce tasks not specified. Estimated from input data size: 11
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0067
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 11
2013-09-17 23:46:53,783 Stage-1 map = 0%, reduce = 0%
2013-09-17 23:47:03,859 Stage-1 map = 2%, reduce = 0%
2013-09-17 23:47:05,878 Stage-1 map = 5%, reduce = 0%
2013-09-17 23:47:07,199 Stage-1 map = 21%, reduce = 0%
2013-09-17 23:47:08,207 Stage-1 map = 24%, reduce = 0%
2013-09-17 23:47:09,214 Stage-1 map = 30%, reduce = 0%
2013-09-17 23:47:10,223 Stage-1 map = 48%, reduce = 0%
2013-09-17 23:47:11,239 Stage-1 map = 50%, reduce = 0%
2013-09-17 23:47:12,265 Stage-1 map = 62%, reduce = 0%, Cumulative CPU 41.92 sec
2013-09-17 23:47:13,282 Stage-1 map = 74%, reduce = 0%, Cumulative CPU 108.6 sec
2013-09-17 23:47:14,295 Stage-1 map = 80%, reduce = 0%, Cumulative CPU 181.12 sec
2013-09-17 23:47:15,307 Stage-1 map = 87%, reduce = 0%, Cumulative CPU 247.1 sec
2013-09-17 23:47:16,313 Stage-1 map = 93%, reduce = 0%, Cumulative CPU 290.6 sec
2013-09-17 23:47:17,320 Stage-1 map = 99%, reduce = 0%, Cumulative CPU 460.07 sec
2013-09-17 23:47:18,326 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 491.41 sec
2013-09-17 23:47:19,332 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 491.41 sec
2013-09-17 23:47:20,338 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 491.41 sec
2013-09-17 23:47:21,345 Stage-1 map = 100%, reduce = 3%, Cumulative CPU 491.41 sec
2013-09-17 23:47:22,352 Stage-1 map = 100%, reduce = 12%, Cumulative CPU 491.41 sec
2013-09-17 23:47:23,359 Stage-1 map = 100%, reduce = 27%, Cumulative CPU 491.41 sec
2013-09-17 23:47:24,369 Stage-1 map = 100%, reduce = 39%, Cumulative CPU 491.41 sec
2013-09-17 23:47:25,377 Stage-1 map = 100%, reduce = 49%, Cumulative CPU 491.41 sec
2013-09-17 23:47:26,389 Stage-1 map = 100%, reduce = 64%, Cumulative CPU 543.54 sec
2013-09-17 23:47:27,397 Stage-1 map = 100%, reduce = 71%, Cumulative CPU 543.54 sec
2013-09-17 23:47:28,404 Stage-1 map = 100%, reduce = 78%, Cumulative CPU 543.54 sec
2013-09-17 23:47:29,411 Stage-1 map = 100%, reduce = 90%, Cumulative CPU 560.73 sec
2013-09-17 23:47:30,418 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 619.13 sec
2013-09-17 23:47:31,425 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 619.13 sec
2013-09-17 23:47:32,432 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 619.13 sec
MapReduce Total cumulative CPU time: 10 minutes 19 seconds 130 msec
Ended Job = job_201309172235_0067
Launching Job 2 out of 2
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0068
Hadoop job information for Stage-2: number of mappers: 2; number of reducers: 1
2013-09-17 23:47:35,954 Stage-2 map = 0%, reduce = 0%
2013-09-17 23:47:42,978 Stage-2 map = 18%, reduce = 0%
2013-09-17 23:47:45,991 Stage-2 map = 37%, reduce = 0%
2013-09-17 23:47:49,003 Stage-2 map = 55%, reduce = 0%
2013-09-17 23:47:52,014 Stage-2 map = 73%, reduce = 0%
2013-09-17 23:47:55,025 Stage-2 map = 82%, reduce = 0%
2013-09-17 23:47:57,034 Stage-2 map = 92%, reduce = 0%, Cumulative CPU 27.07 sec
2013-09-17 23:47:58,040 Stage-2 map = 92%, reduce = 0%, Cumulative CPU 27.07 sec
2013-09-17 23:47:59,046 Stage-2 map = 92%, reduce = 0%, Cumulative CPU 27.07 sec
2013-09-17 23:48:00,051 Stage-2 map = 92%, reduce = 0%, Cumulative CPU 27.07 sec
2013-09-17 23:48:01,056 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 58.79 sec
2013-09-17 23:48:02,060 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 58.79 sec
2013-09-17 23:48:03,066 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 58.79 sec
2013-09-17 23:48:04,072 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 58.79 sec
2013-09-17 23:48:05,077 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 58.79 sec
2013-09-17 23:48:06,082 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 58.79 sec
2013-09-17 23:48:07,087 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 58.79 sec
2013-09-17 23:48:08,092 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 58.79 sec
2013-09-17 23:48:09,096 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 58.79 sec
2013-09-17 23:48:10,102 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 58.79 sec
2013-09-17 23:48:11,107 Stage-2 map = 100%, reduce = 74%, Cumulative CPU 58.79 sec
2013-09-17 23:48:12,112 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 68.73 sec
2013-09-17 23:48:13,119 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 68.73 sec
2013-09-17 23:48:14,124 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 68.73 sec
MapReduce Total cumulative CPU time: 1 minutes 8 seconds 730 msec
Ended Job = job_201309172235_0068
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 11 Cumulative CPU: 619.13 sec HDFS Read: 200023859 HDFS Write: 499984003 SUCCESS
Job 1: Map: 2 Reduce: 1 Cumulative CPU: 68.73 sec HDFS Read: 499987725 HDFS Write: 286 SUCCESS
Total MapReduce CPU Time Spent: 11 minutes 27 seconds 860 msec
OK
Time taken: 91.05 seconds, Fetched: 10 row(s)
hive> quit;
times: 2
query: SELECT SearchEngineID, SearchPhrase, count(*) AS c FROM hits_100m WHERE SearchPhrase != '' GROUP BY SearchEngineID, SearchPhrase ORDER BY c DESC LIMIT 10;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_28179@mturlrep13_201309172348_1562230786.txt
hive> SELECT SearchEngineID, SearchPhrase, count(*) AS c FROM hits_100m WHERE SearchPhrase != '' GROUP BY SearchEngineID, SearchPhrase ORDER BY c DESC LIMIT 10;;
Total MapReduce jobs = 2
Launching Job 1 out of 2
Number of reduce tasks not specified. Estimated from input data size: 11
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0069
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 11
2013-09-17 23:48:23,881 Stage-1 map = 0%, reduce = 0%
2013-09-17 23:48:35,995 Stage-1 map = 2%, reduce = 0%
2013-09-17 23:48:37,001 Stage-1 map = 16%, reduce = 0%
2013-09-17 23:48:38,011 Stage-1 map = 20%, reduce = 0%
2013-09-17 23:48:39,018 Stage-1 map = 24%, reduce = 0%
2013-09-17 23:48:40,024 Stage-1 map = 39%, reduce = 0%
2013-09-17 23:48:41,052 Stage-1 map = 57%, reduce = 0%, Cumulative CPU 10.64 sec
2013-09-17 23:48:42,094 Stage-1 map = 64%, reduce = 0%, Cumulative CPU 99.96 sec
2013-09-17 23:48:43,114 Stage-1 map = 80%, reduce = 0%, Cumulative CPU 158.73 sec
2013-09-17 23:48:44,140 Stage-1 map = 86%, reduce = 0%, Cumulative CPU 239.0 sec
2013-09-17 23:48:45,149 Stage-1 map = 89%, reduce = 0%, Cumulative CPU 313.14 sec
2013-09-17 23:48:46,156 Stage-1 map = 98%, reduce = 0%, Cumulative CPU 453.86 sec
2013-09-17 23:48:47,162 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 503.27 sec
2013-09-17 23:48:48,168 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 503.27 sec
2013-09-17 23:48:49,174 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 503.27 sec
2013-09-17 23:48:50,181 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 503.27 sec
2013-09-17 23:48:51,188 Stage-1 map = 100%, reduce = 12%, Cumulative CPU 503.27 sec
2013-09-17 23:48:52,194 Stage-1 map = 100%, reduce = 27%, Cumulative CPU 503.27 sec
2013-09-17 23:48:53,201 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 503.27 sec
2013-09-17 23:48:54,207 Stage-1 map = 100%, reduce = 45%, Cumulative CPU 503.27 sec
2013-09-17 23:48:55,214 Stage-1 map = 100%, reduce = 61%, Cumulative CPU 503.27 sec
2013-09-17 23:48:56,221 Stage-1 map = 100%, reduce = 67%, Cumulative CPU 503.27 sec
2013-09-17 23:48:57,228 Stage-1 map = 100%, reduce = 76%, Cumulative CPU 503.27 sec
2013-09-17 23:48:58,237 Stage-1 map = 100%, reduce = 90%, Cumulative CPU 548.17 sec
2013-09-17 23:48:59,244 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 606.99 sec
2013-09-17 23:49:00,251 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 630.32 sec
2013-09-17 23:49:01,257 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 630.32 sec
MapReduce Total cumulative CPU time: 10 minutes 30 seconds 320 msec
Ended Job = job_201309172235_0069
Launching Job 2 out of 2
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0070
Hadoop job information for Stage-2: number of mappers: 2; number of reducers: 1
2013-09-17 23:49:04,779 Stage-2 map = 0%, reduce = 0%
2013-09-17 23:49:11,801 Stage-2 map = 18%, reduce = 0%
2013-09-17 23:49:14,812 Stage-2 map = 37%, reduce = 0%
2013-09-17 23:49:17,821 Stage-2 map = 55%, reduce = 0%
2013-09-17 23:49:20,831 Stage-2 map = 73%, reduce = 0%
2013-09-17 23:49:23,839 Stage-2 map = 82%, reduce = 0%
2013-09-17 23:49:26,850 Stage-2 map = 92%, reduce = 0%, Cumulative CPU 52.88 sec
2013-09-17 23:49:27,856 Stage-2 map = 92%, reduce = 0%, Cumulative CPU 52.88 sec
2013-09-17 23:49:28,860 Stage-2 map = 92%, reduce = 0%, Cumulative CPU 52.88 sec
2013-09-17 23:49:29,866 Stage-2 map = 92%, reduce = 0%, Cumulative CPU 52.88 sec
2013-09-17 23:49:32,220 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 60.11 sec
2013-09-17 23:49:33,226 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 60.11 sec
2013-09-17 23:49:34,231 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 60.11 sec
2013-09-17 23:49:35,237 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 60.11 sec
2013-09-17 23:49:36,242 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 60.11 sec
2013-09-17 23:49:37,248 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 60.11 sec
2013-09-17 23:49:38,253 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 60.11 sec
2013-09-17 23:49:39,259 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 60.11 sec
2013-09-17 23:49:40,265 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 60.11 sec
2013-09-17 23:49:41,270 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 70.2 sec
2013-09-17 23:49:42,276 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 70.2 sec
2013-09-17 23:49:43,281 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 70.2 sec
MapReduce Total cumulative CPU time: 1 minutes 10 seconds 200 msec
Ended Job = job_201309172235_0070
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 11 Cumulative CPU: 630.32 sec HDFS Read: 200023859 HDFS Write: 499984003 SUCCESS
Job 1: Map: 2 Reduce: 1 Cumulative CPU: 70.2 sec HDFS Read: 499987725 HDFS Write: 286 SUCCESS
Total MapReduce CPU Time Spent: 11 minutes 40 seconds 520 msec
OK
Time taken: 87.267 seconds, Fetched: 10 row(s)
hive> quit;
times: 3
query: SELECT SearchEngineID, SearchPhrase, count(*) AS c FROM hits_100m WHERE SearchPhrase != '' GROUP BY SearchEngineID, SearchPhrase ORDER BY c DESC LIMIT 10;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_2628@mturlrep13_201309172349_1108098215.txt
hive> SELECT SearchEngineID, SearchPhrase, count(*) AS c FROM hits_100m WHERE SearchPhrase != '' GROUP BY SearchEngineID, SearchPhrase ORDER BY c DESC LIMIT 10;;
Total MapReduce jobs = 2
Launching Job 1 out of 2
Number of reduce tasks not specified. Estimated from input data size: 11
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0071
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 11
2013-09-17 23:49:52,985 Stage-1 map = 0%, reduce = 0%
2013-09-17 23:50:05,128 Stage-1 map = 2%, reduce = 0%
2013-09-17 23:50:06,133 Stage-1 map = 15%, reduce = 0%
2013-09-17 23:50:07,151 Stage-1 map = 22%, reduce = 0%
2013-09-17 23:50:08,158 Stage-1 map = 23%, reduce = 0%
2013-09-17 23:50:09,175 Stage-1 map = 50%, reduce = 0%
2013-09-17 23:50:10,194 Stage-1 map = 56%, reduce = 0%, Cumulative CPU 31.85 sec
2013-09-17 23:50:11,210 Stage-1 map = 63%, reduce = 0%, Cumulative CPU 65.05 sec
2013-09-17 23:50:12,225 Stage-1 map = 77%, reduce = 0%, Cumulative CPU 147.44 sec
2013-09-17 23:50:13,259 Stage-1 map = 83%, reduce = 0%, Cumulative CPU 212.15 sec
2013-09-17 23:50:14,277 Stage-1 map = 87%, reduce = 0%, Cumulative CPU 269.6 sec
2013-09-17 23:50:15,286 Stage-1 map = 97%, reduce = 0%, Cumulative CPU 409.48 sec
2013-09-17 23:50:16,293 Stage-1 map = 99%, reduce = 0%, Cumulative CPU 490.26 sec
2013-09-17 23:50:17,299 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 507.73 sec
2013-09-17 23:50:18,304 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 507.73 sec
2013-09-17 23:50:19,311 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 507.73 sec
2013-09-17 23:50:20,317 Stage-1 map = 100%, reduce = 6%, Cumulative CPU 507.73 sec
2013-09-17 23:50:21,323 Stage-1 map = 100%, reduce = 27%, Cumulative CPU 507.73 sec
2013-09-17 23:50:22,331 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 507.73 sec
2013-09-17 23:50:23,337 Stage-1 map = 100%, reduce = 39%, Cumulative CPU 507.73 sec
2013-09-17 23:50:24,344 Stage-1 map = 100%, reduce = 58%, Cumulative CPU 507.73 sec
2013-09-17 23:50:25,351 Stage-1 map = 100%, reduce = 67%, Cumulative CPU 507.73 sec
2013-09-17 23:50:26,357 Stage-1 map = 100%, reduce = 71%, Cumulative CPU 507.73 sec
2013-09-17 23:50:27,367 Stage-1 map = 100%, reduce = 86%, Cumulative CPU 592.09 sec
2013-09-17 23:50:28,374 Stage-1 map = 100%, reduce = 98%, Cumulative CPU 620.73 sec
2013-09-17 23:50:29,381 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 636.3 sec
2013-09-17 23:50:30,387 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 636.3 sec
MapReduce Total cumulative CPU time: 10 minutes 36 seconds 300 msec
Ended Job = job_201309172235_0071
Launching Job 2 out of 2
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0072
Hadoop job information for Stage-2: number of mappers: 2; number of reducers: 1
2013-09-17 23:50:33,867 Stage-2 map = 0%, reduce = 0%
2013-09-17 23:50:40,890 Stage-2 map = 18%, reduce = 0%
2013-09-17 23:50:43,901 Stage-2 map = 37%, reduce = 0%
2013-09-17 23:50:46,911 Stage-2 map = 55%, reduce = 0%
2013-09-17 23:50:49,921 Stage-2 map = 73%, reduce = 0%
2013-09-17 23:50:53,934 Stage-2 map = 82%, reduce = 0%
2013-09-17 23:50:55,943 Stage-2 map = 92%, reduce = 0%, Cumulative CPU 27.87 sec
2013-09-17 23:50:56,949 Stage-2 map = 92%, reduce = 0%, Cumulative CPU 27.87 sec
2013-09-17 23:50:57,955 Stage-2 map = 92%, reduce = 0%, Cumulative CPU 27.87 sec
2013-09-17 23:50:59,680 Stage-2 map = 92%, reduce = 0%, Cumulative CPU 27.87 sec
2013-09-17 23:51:00,685 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 61.27 sec
2013-09-17 23:51:01,690 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 61.27 sec
2013-09-17 23:51:02,695 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 61.27 sec
2013-09-17 23:51:03,700 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 61.27 sec
2013-09-17 23:51:04,705 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 61.27 sec
2013-09-17 23:51:05,710 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 61.27 sec
2013-09-17 23:51:06,715 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 61.27 sec
2013-09-17 23:51:07,719 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 61.27 sec
2013-09-17 23:51:08,725 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 61.27 sec
2013-09-17 23:51:09,730 Stage-2 map = 100%, reduce = 74%, Cumulative CPU 61.27 sec
2013-09-17 23:51:10,735 Stage-2 map = 100%, reduce = 74%, Cumulative CPU 61.27 sec
2013-09-17 23:51:11,740 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 71.17 sec
2013-09-17 23:51:12,746 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 71.17 sec
MapReduce Total cumulative CPU time: 1 minutes 11 seconds 170 msec
Ended Job = job_201309172235_0072
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 11 Cumulative CPU: 636.3 sec HDFS Read: 200023859 HDFS Write: 499984003 SUCCESS
Job 1: Map: 2 Reduce: 1 Cumulative CPU: 71.17 sec HDFS Read: 499987725 HDFS Write: 286 SUCCESS
Total MapReduce CPU Time Spent: 11 minutes 47 seconds 470 msec
OK
Time taken: 87.731 seconds, Fetched: 10 row(s)
hive> quit;
-- агрегация по числу и строке, большое количество ключей.;
times: 1
query: SELECT UserID, count(*) AS c FROM hits_100m GROUP BY UserID ORDER BY c DESC LIMIT 10;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_10539@mturlrep13_201309172351_1364032363.txt
hive> SELECT UserID, count(*) AS c FROM hits_100m GROUP BY UserID ORDER BY c DESC LIMIT 10;;
Total MapReduce jobs = 2
Launching Job 1 out of 2
Number of reduce tasks not specified. Estimated from input data size: 11
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0073
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 11
2013-09-17 23:51:32,893 Stage-1 map = 0%, reduce = 0%
2013-09-17 23:51:43,007 Stage-1 map = 1%, reduce = 0%
2013-09-17 23:51:45,021 Stage-1 map = 4%, reduce = 0%
2013-09-17 23:51:47,070 Stage-1 map = 11%, reduce = 0%
2013-09-17 23:51:48,076 Stage-1 map = 14%, reduce = 0%
2013-09-17 23:51:50,095 Stage-1 map = 33%, reduce = 0%
2013-09-17 23:51:51,260 Stage-1 map = 39%, reduce = 0%
2013-09-17 23:51:53,285 Stage-1 map = 51%, reduce = 0%
2013-09-17 23:51:54,299 Stage-1 map = 58%, reduce = 0%
2013-09-17 23:51:55,943 Stage-1 map = 59%, reduce = 0%, Cumulative CPU 13.64 sec
2013-09-17 23:51:56,974 Stage-1 map = 73%, reduce = 0%, Cumulative CPU 25.01 sec
2013-09-17 23:51:57,984 Stage-1 map = 75%, reduce = 0%, Cumulative CPU 38.32 sec
2013-09-17 23:51:58,990 Stage-1 map = 76%, reduce = 0%, Cumulative CPU 80.19 sec
2013-09-17 23:51:59,996 Stage-1 map = 83%, reduce = 0%, Cumulative CPU 80.19 sec
2013-09-17 23:52:01,007 Stage-1 map = 84%, reduce = 0%, Cumulative CPU 95.79 sec
2013-09-17 23:52:02,036 Stage-1 map = 86%, reduce = 0%, Cumulative CPU 193.09 sec
2013-09-17 23:52:03,054 Stage-1 map = 90%, reduce = 0%, Cumulative CPU 243.99 sec
2013-09-17 23:52:04,061 Stage-1 map = 95%, reduce = 0%, Cumulative CPU 426.96 sec
2013-09-17 23:52:05,067 Stage-1 map = 95%, reduce = 0%, Cumulative CPU 468.05 sec
2013-09-17 23:52:06,074 Stage-1 map = 97%, reduce = 0%, Cumulative CPU 487.46 sec
2013-09-17 23:52:07,080 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 685.94 sec
2013-09-17 23:52:08,087 Stage-1 map = 100%, reduce = 2%, Cumulative CPU 685.94 sec
2013-09-17 23:52:09,093 Stage-1 map = 100%, reduce = 12%, Cumulative CPU 685.94 sec
2013-09-17 23:52:10,100 Stage-1 map = 100%, reduce = 21%, Cumulative CPU 685.94 sec
2013-09-17 23:52:11,107 Stage-1 map = 100%, reduce = 25%, Cumulative CPU 685.94 sec
2013-09-17 23:52:12,114 Stage-1 map = 100%, reduce = 27%, Cumulative CPU 685.94 sec
2013-09-17 23:52:13,122 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 685.94 sec
2013-09-17 23:52:14,128 Stage-1 map = 100%, reduce = 40%, Cumulative CPU 685.94 sec
2013-09-17 23:52:15,135 Stage-1 map = 100%, reduce = 51%, Cumulative CPU 685.94 sec
2013-09-17 23:52:16,141 Stage-1 map = 100%, reduce = 72%, Cumulative CPU 685.94 sec
2013-09-17 23:52:17,148 Stage-1 map = 100%, reduce = 75%, Cumulative CPU 685.94 sec
2013-09-17 23:52:18,153 Stage-1 map = 100%, reduce = 79%, Cumulative CPU 685.94 sec
2013-09-17 23:52:19,170 Stage-1 map = 100%, reduce = 88%, Cumulative CPU 700.91 sec
2013-09-17 23:52:20,177 Stage-1 map = 100%, reduce = 95%, Cumulative CPU 810.21 sec
2013-09-17 23:52:21,183 Stage-1 map = 100%, reduce = 96%, Cumulative CPU 825.67 sec
2013-09-17 23:52:22,188 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 825.67 sec
2013-09-17 23:52:23,194 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 855.09 sec
2013-09-17 23:52:24,198 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 855.09 sec
MapReduce Total cumulative CPU time: 14 minutes 15 seconds 90 msec
Ended Job = job_201309172235_0073
Launching Job 2 out of 2
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0074
Hadoop job information for Stage-2: number of mappers: 2; number of reducers: 1
2013-09-17 23:52:27,714 Stage-2 map = 0%, reduce = 0%
2013-09-17 23:52:40,754 Stage-2 map = 18%, reduce = 0%
2013-09-17 23:52:48,779 Stage-2 map = 27%, reduce = 0%
2013-09-17 23:52:49,783 Stage-2 map = 37%, reduce = 0%
2013-09-17 23:52:52,793 Stage-2 map = 45%, reduce = 0%
2013-09-17 23:52:55,803 Stage-2 map = 55%, reduce = 0%
2013-09-17 23:53:01,889 Stage-2 map = 73%, reduce = 0%
2013-09-17 23:53:07,908 Stage-2 map = 82%, reduce = 0%
2013-09-17 23:53:13,929 Stage-2 map = 92%, reduce = 0%, Cumulative CPU 55.32 sec
2013-09-17 23:53:14,935 Stage-2 map = 92%, reduce = 0%, Cumulative CPU 55.32 sec
2013-09-17 23:53:15,940 Stage-2 map = 92%, reduce = 0%, Cumulative CPU 55.32 sec
2013-09-17 23:53:16,946 Stage-2 map = 92%, reduce = 0%, Cumulative CPU 55.32 sec
2013-09-17 23:53:17,951 Stage-2 map = 92%, reduce = 0%, Cumulative CPU 55.32 sec
2013-09-17 23:53:18,956 Stage-2 map = 92%, reduce = 0%, Cumulative CPU 55.32 sec
2013-09-17 23:53:19,961 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 118.9 sec
2013-09-17 23:53:20,966 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 118.9 sec
2013-09-17 23:53:21,971 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 118.9 sec
2013-09-17 23:53:22,975 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 118.9 sec
2013-09-17 23:53:23,979 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 118.9 sec
2013-09-17 23:53:24,983 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 118.9 sec
2013-09-17 23:53:25,987 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 118.9 sec
2013-09-17 23:53:26,992 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 123.85 sec
2013-09-17 23:53:27,996 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 123.85 sec
2013-09-17 23:53:29,001 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 123.85 sec
2013-09-17 23:53:30,005 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 123.85 sec
2013-09-17 23:53:31,010 Stage-2 map = 100%, reduce = 68%, Cumulative CPU 123.85 sec
2013-09-17 23:53:32,014 Stage-2 map = 100%, reduce = 68%, Cumulative CPU 123.85 sec
2013-09-17 23:53:33,019 Stage-2 map = 100%, reduce = 68%, Cumulative CPU 123.85 sec
2013-09-17 23:53:34,029 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 133.7 sec
2013-09-17 23:53:35,034 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 133.7 sec
MapReduce Total cumulative CPU time: 2 minutes 13 seconds 700 msec
Ended Job = job_201309172235_0074
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 11 Cumulative CPU: 855.09 sec HDFS Read: 663537300 HDFS Write: 480155183 SUCCESS
Job 1: Map: 2 Reduce: 1 Cumulative CPU: 133.7 sec HDFS Read: 480158905 HDFS Write: 251 SUCCESS
Total MapReduce CPU Time Spent: 16 minutes 28 seconds 790 msec
OK
Time taken: 132.697 seconds, Fetched: 10 row(s)
hive> quit;
times: 2
query: SELECT UserID, count(*) AS c FROM hits_100m GROUP BY UserID ORDER BY c DESC LIMIT 10;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_17435@mturlrep13_201309172353_888901844.txt
hive> SELECT UserID, count(*) AS c FROM hits_100m GROUP BY UserID ORDER BY c DESC LIMIT 10;;
Total MapReduce jobs = 2
Launching Job 1 out of 2
Number of reduce tasks not specified. Estimated from input data size: 11
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0075
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 11
2013-09-17 23:53:45,965 Stage-1 map = 0%, reduce = 0%
2013-09-17 23:53:57,063 Stage-1 map = 1%, reduce = 0%
2013-09-17 23:53:58,076 Stage-1 map = 9%, reduce = 0%
2013-09-17 23:53:59,095 Stage-1 map = 20%, reduce = 0%
2013-09-17 23:54:00,108 Stage-1 map = 23%, reduce = 0%
2013-09-17 23:54:01,121 Stage-1 map = 37%, reduce = 0%
2013-09-17 23:54:02,132 Stage-1 map = 58%, reduce = 0%
2013-09-17 23:54:03,151 Stage-1 map = 59%, reduce = 0%
2013-09-17 23:54:04,163 Stage-1 map = 63%, reduce = 0%, Cumulative CPU 10.96 sec
2013-09-17 23:54:05,184 Stage-1 map = 74%, reduce = 0%, Cumulative CPU 51.58 sec
2013-09-17 23:54:06,196 Stage-1 map = 77%, reduce = 0%, Cumulative CPU 65.4 sec
2013-09-17 23:54:07,205 Stage-1 map = 79%, reduce = 0%, Cumulative CPU 123.94 sec
2013-09-17 23:54:08,310 Stage-1 map = 83%, reduce = 0%, Cumulative CPU 137.81 sec
2013-09-17 23:54:09,332 Stage-1 map = 85%, reduce = 0%, Cumulative CPU 234.2 sec
2013-09-17 23:54:10,342 Stage-1 map = 87%, reduce = 0%, Cumulative CPU 283.15 sec
2013-09-17 23:54:11,350 Stage-1 map = 92%, reduce = 0%, Cumulative CPU 399.5 sec
2013-09-17 23:54:12,356 Stage-1 map = 93%, reduce = 0%, Cumulative CPU 459.78 sec
2013-09-17 23:54:13,365 Stage-1 map = 95%, reduce = 0%, Cumulative CPU 479.7 sec
2013-09-17 23:54:14,372 Stage-1 map = 97%, reduce = 0%, Cumulative CPU 479.7 sec
2013-09-17 23:54:15,378 Stage-1 map = 98%, reduce = 5%, Cumulative CPU 553.18 sec
2013-09-17 23:54:16,383 Stage-1 map = 99%, reduce = 17%, Cumulative CPU 654.99 sec
2013-09-17 23:54:17,388 Stage-1 map = 100%, reduce = 23%, Cumulative CPU 681.14 sec
2013-09-17 23:54:18,394 Stage-1 map = 100%, reduce = 27%, Cumulative CPU 681.14 sec
2013-09-17 23:54:19,399 Stage-1 map = 100%, reduce = 27%, Cumulative CPU 681.14 sec
2013-09-17 23:54:20,405 Stage-1 map = 100%, reduce = 27%, Cumulative CPU 681.14 sec
2013-09-17 23:54:21,411 Stage-1 map = 100%, reduce = 29%, Cumulative CPU 681.14 sec
2013-09-17 23:54:22,417 Stage-1 map = 100%, reduce = 30%, Cumulative CPU 681.14 sec
2013-09-17 23:54:23,423 Stage-1 map = 100%, reduce = 30%, Cumulative CPU 681.14 sec
2013-09-17 23:54:24,430 Stage-1 map = 100%, reduce = 39%, Cumulative CPU 681.14 sec
2013-09-17 23:54:25,442 Stage-1 map = 100%, reduce = 56%, Cumulative CPU 681.14 sec
2013-09-17 23:54:26,448 Stage-1 map = 100%, reduce = 67%, Cumulative CPU 681.14 sec
2013-09-17 23:54:27,456 Stage-1 map = 100%, reduce = 70%, Cumulative CPU 758.79 sec
2013-09-17 23:54:28,462 Stage-1 map = 100%, reduce = 76%, Cumulative CPU 758.79 sec
2013-09-17 23:54:29,468 Stage-1 map = 100%, reduce = 80%, Cumulative CPU 758.79 sec
2013-09-17 23:54:30,475 Stage-1 map = 100%, reduce = 84%, Cumulative CPU 758.79 sec
2013-09-17 23:54:31,481 Stage-1 map = 100%, reduce = 92%, Cumulative CPU 792.19 sec
2013-09-17 23:54:32,487 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 841.44 sec
2013-09-17 23:54:33,493 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 849.83 sec
2013-09-17 23:54:34,499 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 849.83 sec
MapReduce Total cumulative CPU time: 14 minutes 9 seconds 830 msec
Ended Job = job_201309172235_0075
Launching Job 2 out of 2
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0076
Hadoop job information for Stage-2: number of mappers: 2; number of reducers: 1
2013-09-17 23:54:36,978 Stage-2 map = 0%, reduce = 0%
2013-09-17 23:54:51,026 Stage-2 map = 18%, reduce = 0%
2013-09-17 23:54:57,047 Stage-2 map = 37%, reduce = 0%
2013-09-17 23:55:04,073 Stage-2 map = 47%, reduce = 0%
2013-09-17 23:55:07,084 Stage-2 map = 55%, reduce = 0%
2013-09-17 23:55:10,096 Stage-2 map = 65%, reduce = 0%
2013-09-17 23:55:14,109 Stage-2 map = 73%, reduce = 0%
2013-09-17 23:55:20,130 Stage-2 map = 82%, reduce = 0%
2013-09-17 23:55:23,142 Stage-2 map = 92%, reduce = 0%, Cumulative CPU 54.08 sec
2013-09-17 23:55:24,147 Stage-2 map = 92%, reduce = 0%, Cumulative CPU 54.08 sec
2013-09-17 23:55:25,151 Stage-2 map = 92%, reduce = 0%, Cumulative CPU 54.08 sec
2013-09-17 23:55:26,156 Stage-2 map = 92%, reduce = 0%, Cumulative CPU 54.08 sec
2013-09-17 23:55:27,160 Stage-2 map = 92%, reduce = 0%, Cumulative CPU 112.2 sec
2013-09-17 23:55:28,165 Stage-2 map = 92%, reduce = 0%, Cumulative CPU 112.2 sec
2013-09-17 23:55:29,170 Stage-2 map = 92%, reduce = 0%, Cumulative CPU 112.2 sec
2013-09-17 23:55:30,175 Stage-2 map = 92%, reduce = 0%, Cumulative CPU 112.2 sec
2013-09-17 23:55:31,180 Stage-2 map = 92%, reduce = 0%, Cumulative CPU 112.2 sec
2013-09-17 23:55:32,185 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 118.52 sec
2013-09-17 23:55:33,190 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 118.52 sec
2013-09-17 23:55:34,194 Stage-2 map = 100%, reduce = 17%, Cumulative CPU 118.52 sec
2013-09-17 23:55:35,198 Stage-2 map = 100%, reduce = 17%, Cumulative CPU 118.52 sec
2013-09-17 23:55:36,202 Stage-2 map = 100%, reduce = 17%, Cumulative CPU 118.52 sec
2013-09-17 23:55:37,207 Stage-2 map = 100%, reduce = 17%, Cumulative CPU 118.52 sec
2013-09-17 23:55:38,212 Stage-2 map = 100%, reduce = 17%, Cumulative CPU 118.52 sec
2013-09-17 23:55:39,217 Stage-2 map = 100%, reduce = 17%, Cumulative CPU 118.52 sec
2013-09-17 23:55:40,222 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 118.52 sec
2013-09-17 23:55:41,227 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 118.52 sec
2013-09-17 23:55:42,233 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 118.52 sec
2013-09-17 23:55:43,248 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 118.52 sec
2013-09-17 23:55:44,252 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 118.52 sec
2013-09-17 23:55:45,257 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 118.52 sec
2013-09-17 23:55:46,262 Stage-2 map = 100%, reduce = 80%, Cumulative CPU 118.52 sec
2013-09-17 23:55:47,267 Stage-2 map = 100%, reduce = 80%, Cumulative CPU 118.52 sec
2013-09-17 23:55:48,272 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 133.0 sec
2013-09-17 23:55:49,278 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 133.0 sec
MapReduce Total cumulative CPU time: 2 minutes 13 seconds 0 msec
Ended Job = job_201309172235_0076
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 11 Cumulative CPU: 849.83 sec HDFS Read: 663537300 HDFS Write: 480155183 SUCCESS
Job 1: Map: 2 Reduce: 1 Cumulative CPU: 133.0 sec HDFS Read: 480158905 HDFS Write: 251 SUCCESS
Total MapReduce CPU Time Spent: 16 minutes 22 seconds 830 msec
OK
Time taken: 132.306 seconds, Fetched: 10 row(s)
hive> quit;
times: 3
query: SELECT UserID, count(*) AS c FROM hits_100m GROUP BY UserID ORDER BY c DESC LIMIT 10;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_25039@mturlrep13_201309172355_777652129.txt
hive> SELECT UserID, count(*) AS c FROM hits_100m GROUP BY UserID ORDER BY c DESC LIMIT 10;;
Total MapReduce jobs = 2
Launching Job 1 out of 2
Number of reduce tasks not specified. Estimated from input data size: 11
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0077
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 11
2013-09-17 23:55:58,731 Stage-1 map = 0%, reduce = 0%
2013-09-17 23:56:11,863 Stage-1 map = 12%, reduce = 0%
2013-09-17 23:56:12,877 Stage-1 map = 20%, reduce = 0%
2013-09-17 23:56:13,887 Stage-1 map = 22%, reduce = 0%
2013-09-17 23:56:14,900 Stage-1 map = 40%, reduce = 0%
2013-09-17 23:56:15,915 Stage-1 map = 57%, reduce = 0%
2013-09-17 23:56:16,938 Stage-1 map = 58%, reduce = 0%, Cumulative CPU 10.56 sec
2013-09-17 23:56:17,950 Stage-1 map = 60%, reduce = 0%, Cumulative CPU 10.56 sec
2013-09-17 23:56:18,963 Stage-1 map = 74%, reduce = 0%, Cumulative CPU 10.56 sec
2013-09-17 23:56:19,993 Stage-1 map = 76%, reduce = 0%, Cumulative CPU 52.39 sec
2013-09-17 23:56:20,999 Stage-1 map = 78%, reduce = 0%, Cumulative CPU 109.68 sec
2013-09-17 23:56:22,006 Stage-1 map = 83%, reduce = 0%, Cumulative CPU 171.28 sec
2013-09-17 23:56:23,012 Stage-1 map = 85%, reduce = 0%, Cumulative CPU 218.43 sec
2013-09-17 23:56:24,036 Stage-1 map = 86%, reduce = 0%, Cumulative CPU 285.81 sec
2013-09-17 23:56:25,051 Stage-1 map = 91%, reduce = 0%, Cumulative CPU 343.66 sec
2013-09-17 23:56:26,057 Stage-1 map = 94%, reduce = 0%, Cumulative CPU 482.51 sec
2013-09-17 23:56:27,063 Stage-1 map = 94%, reduce = 0%, Cumulative CPU 635.88 sec
2013-09-17 23:56:28,070 Stage-1 map = 97%, reduce = 0%, Cumulative CPU 635.88 sec
2013-09-17 23:56:29,077 Stage-1 map = 98%, reduce = 0%, Cumulative CPU 651.74 sec
2013-09-17 23:56:30,084 Stage-1 map = 100%, reduce = 7%, Cumulative CPU 682.92 sec
2013-09-17 23:56:31,091 Stage-1 map = 100%, reduce = 19%, Cumulative CPU 682.92 sec
2013-09-17 23:56:32,097 Stage-1 map = 100%, reduce = 31%, Cumulative CPU 682.92 sec
2013-09-17 23:56:33,104 Stage-1 map = 100%, reduce = 31%, Cumulative CPU 682.92 sec
2013-09-17 23:56:34,110 Stage-1 map = 100%, reduce = 31%, Cumulative CPU 682.92 sec
2013-09-17 23:56:35,117 Stage-1 map = 100%, reduce = 40%, Cumulative CPU 682.92 sec
2013-09-17 23:56:36,123 Stage-1 map = 100%, reduce = 40%, Cumulative CPU 682.92 sec
2013-09-17 23:56:37,129 Stage-1 map = 100%, reduce = 40%, Cumulative CPU 682.92 sec
2013-09-17 23:56:38,136 Stage-1 map = 100%, reduce = 43%, Cumulative CPU 682.92 sec
2013-09-17 23:56:39,142 Stage-1 map = 100%, reduce = 54%, Cumulative CPU 682.92 sec
2013-09-17 23:56:40,150 Stage-1 map = 100%, reduce = 67%, Cumulative CPU 682.92 sec
2013-09-17 23:56:41,155 Stage-1 map = 100%, reduce = 74%, Cumulative CPU 682.92 sec
2013-09-17 23:56:42,163 Stage-1 map = 100%, reduce = 79%, Cumulative CPU 725.63 sec
2013-09-17 23:56:43,170 Stage-1 map = 100%, reduce = 84%, Cumulative CPU 725.63 sec
2013-09-17 23:56:44,176 Stage-1 map = 100%, reduce = 86%, Cumulative CPU 725.63 sec
2013-09-17 23:56:45,182 Stage-1 map = 100%, reduce = 91%, Cumulative CPU 755.1 sec
2013-09-17 23:56:46,188 Stage-1 map = 100%, reduce = 98%, Cumulative CPU 785.47 sec
2013-09-17 23:56:47,195 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 845.55 sec
2013-09-17 23:56:48,201 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 845.55 sec
2013-09-17 23:56:49,207 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 845.55 sec
MapReduce Total cumulative CPU time: 14 minutes 5 seconds 550 msec
Ended Job = job_201309172235_0077
Launching Job 2 out of 2
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0078
Hadoop job information for Stage-2: number of mappers: 2; number of reducers: 1
2013-09-17 23:56:51,716 Stage-2 map = 0%, reduce = 0%
2013-09-17 23:57:05,763 Stage-2 map = 18%, reduce = 0%
2013-09-17 23:57:11,783 Stage-2 map = 37%, reduce = 0%
2013-09-17 23:57:17,803 Stage-2 map = 55%, reduce = 0%
2013-09-17 23:57:23,823 Stage-2 map = 73%, reduce = 0%
2013-09-17 23:57:29,845 Stage-2 map = 82%, reduce = 0%, Cumulative CPU 87.94 sec
2013-09-17 23:57:30,850 Stage-2 map = 82%, reduce = 0%, Cumulative CPU 87.94 sec
2013-09-17 23:57:31,855 Stage-2 map = 82%, reduce = 0%, Cumulative CPU 87.94 sec
2013-09-17 23:57:32,860 Stage-2 map = 82%, reduce = 0%, Cumulative CPU 87.94 sec
2013-09-17 23:57:33,865 Stage-2 map = 82%, reduce = 0%, Cumulative CPU 87.94 sec
2013-09-17 23:57:34,870 Stage-2 map = 82%, reduce = 0%, Cumulative CPU 87.94 sec
2013-09-17 23:57:35,876 Stage-2 map = 92%, reduce = 0%, Cumulative CPU 97.95 sec
2013-09-17 23:57:36,881 Stage-2 map = 92%, reduce = 0%, Cumulative CPU 97.95 sec
2013-09-17 23:57:37,887 Stage-2 map = 92%, reduce = 0%, Cumulative CPU 97.95 sec
2013-09-17 23:57:38,893 Stage-2 map = 92%, reduce = 0%, Cumulative CPU 97.95 sec
2013-09-17 23:57:39,898 Stage-2 map = 92%, reduce = 0%, Cumulative CPU 97.95 sec
2013-09-17 23:57:40,903 Stage-2 map = 92%, reduce = 0%, Cumulative CPU 97.95 sec
2013-09-17 23:57:41,908 Stage-2 map = 92%, reduce = 0%, Cumulative CPU 97.95 sec
2013-09-17 23:57:42,914 Stage-2 map = 92%, reduce = 0%, Cumulative CPU 97.95 sec
2013-09-17 23:57:43,919 Stage-2 map = 92%, reduce = 0%, Cumulative CPU 97.95 sec
2013-09-17 23:57:44,924 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 118.7 sec
2013-09-17 23:57:45,929 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 118.7 sec
2013-09-17 23:57:46,934 Stage-2 map = 100%, reduce = 17%, Cumulative CPU 118.7 sec
2013-09-17 23:57:47,939 Stage-2 map = 100%, reduce = 17%, Cumulative CPU 118.7 sec
2013-09-17 23:57:48,944 Stage-2 map = 100%, reduce = 17%, Cumulative CPU 118.7 sec
2013-09-17 23:57:49,948 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 118.7 sec
2013-09-17 23:57:50,960 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 118.7 sec
2013-09-17 23:57:51,965 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 118.7 sec
2013-09-17 23:57:52,969 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 118.7 sec
2013-09-17 23:57:53,974 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 118.7 sec
2013-09-17 23:57:54,978 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 118.7 sec
2013-09-17 23:57:55,982 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 118.7 sec
2013-09-17 23:57:56,986 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 118.7 sec
2013-09-17 23:57:57,991 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 118.7 sec
2013-09-17 23:57:58,995 Stage-2 map = 100%, reduce = 80%, Cumulative CPU 118.7 sec
2013-09-17 23:58:00,000 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 133.21 sec
2013-09-17 23:58:01,005 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 133.21 sec
2013-09-17 23:58:02,010 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 133.21 sec
MapReduce Total cumulative CPU time: 2 minutes 13 seconds 210 msec
Ended Job = job_201309172235_0078
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 11 Cumulative CPU: 845.55 sec HDFS Read: 663537300 HDFS Write: 480155183 SUCCESS
Job 1: Map: 2 Reduce: 1 Cumulative CPU: 133.21 sec HDFS Read: 480158905 HDFS Write: 251 SUCCESS
Total MapReduce CPU Time Spent: 16 minutes 18 seconds 760 msec
OK
Time taken: 130.91 seconds, Fetched: 10 row(s)
hive> quit;
-- агрегация по очень большому количеству ключей, может не хватить оперативки.;
times: 1
query: SELECT UserID, SearchPhrase, count(*) AS c FROM hits_100m GROUP BY UserID, SearchPhrase ORDER BY c DESC LIMIT 10;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_31945@mturlrep13_201309172358_1169432593.txt
hive> SELECT UserID, SearchPhrase, count(*) AS c FROM hits_100m GROUP BY UserID, SearchPhrase ORDER BY c DESC LIMIT 10;;
Total MapReduce jobs = 2
Launching Job 1 out of 2
Number of reduce tasks not specified. Estimated from input data size: 11
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0079
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 11
2013-09-17 23:58:22,443 Stage-1 map = 0%, reduce = 0%
2013-09-17 23:58:32,503 Stage-1 map = 1%, reduce = 0%
2013-09-17 23:58:36,545 Stage-1 map = 12%, reduce = 0%
2013-09-17 23:58:39,582 Stage-1 map = 18%, reduce = 0%
2013-09-17 23:58:41,277 Stage-1 map = 26%, reduce = 0%
2013-09-17 23:58:43,786 Stage-1 map = 40%, reduce = 0%
2013-09-17 23:58:46,818 Stage-1 map = 59%, reduce = 0%, Cumulative CPU 15.02 sec
2013-09-17 23:58:47,825 Stage-1 map = 59%, reduce = 0%, Cumulative CPU 15.02 sec
2013-09-17 23:58:48,836 Stage-1 map = 59%, reduce = 0%, Cumulative CPU 15.02 sec
2013-09-17 23:58:49,842 Stage-1 map = 70%, reduce = 0%, Cumulative CPU 31.76 sec
2013-09-17 23:58:50,856 Stage-1 map = 74%, reduce = 0%, Cumulative CPU 31.76 sec
2013-09-17 23:58:51,862 Stage-1 map = 75%, reduce = 0%, Cumulative CPU 83.8 sec
2013-09-17 23:58:52,870 Stage-1 map = 77%, reduce = 0%, Cumulative CPU 99.52 sec
2013-09-17 23:58:53,884 Stage-1 map = 81%, reduce = 0%, Cumulative CPU 99.52 sec
2013-09-17 23:58:54,894 Stage-1 map = 81%, reduce = 0%, Cumulative CPU 117.13 sec
2013-09-17 23:58:55,909 Stage-1 map = 84%, reduce = 0%, Cumulative CPU 213.91 sec
2013-09-17 23:58:56,924 Stage-1 map = 88%, reduce = 0%, Cumulative CPU 324.7 sec
2013-09-17 23:58:57,930 Stage-1 map = 91%, reduce = 0%, Cumulative CPU 449.36 sec
2013-09-17 23:58:58,936 Stage-1 map = 92%, reduce = 0%, Cumulative CPU 491.93 sec
2013-09-17 23:58:59,943 Stage-1 map = 96%, reduce = 0%, Cumulative CPU 542.45 sec
2013-09-17 23:59:00,949 Stage-1 map = 97%, reduce = 0%, Cumulative CPU 591.47 sec
2013-09-17 23:59:01,955 Stage-1 map = 97%, reduce = 3%, Cumulative CPU 620.12 sec
2013-09-17 23:59:02,961 Stage-1 map = 99%, reduce = 10%, Cumulative CPU 796.58 sec
2013-09-17 23:59:03,967 Stage-1 map = 99%, reduce = 10%, Cumulative CPU 796.58 sec
2013-09-17 23:59:04,972 Stage-1 map = 100%, reduce = 12%, Cumulative CPU 827.74 sec
2013-09-17 23:59:05,980 Stage-1 map = 100%, reduce = 17%, Cumulative CPU 827.74 sec
2013-09-17 23:59:06,987 Stage-1 map = 100%, reduce = 26%, Cumulative CPU 827.74 sec
2013-09-17 23:59:07,994 Stage-1 map = 100%, reduce = 30%, Cumulative CPU 827.74 sec
2013-09-17 23:59:09,000 Stage-1 map = 100%, reduce = 32%, Cumulative CPU 827.74 sec
2013-09-17 23:59:10,007 Stage-1 map = 100%, reduce = 32%, Cumulative CPU 827.74 sec
2013-09-17 23:59:11,013 Stage-1 map = 100%, reduce = 32%, Cumulative CPU 827.74 sec
2013-09-17 23:59:12,020 Stage-1 map = 100%, reduce = 32%, Cumulative CPU 827.74 sec
2013-09-17 23:59:13,027 Stage-1 map = 100%, reduce = 49%, Cumulative CPU 827.74 sec
2013-09-17 23:59:14,033 Stage-1 map = 100%, reduce = 56%, Cumulative CPU 827.74 sec
2013-09-17 23:59:15,039 Stage-1 map = 100%, reduce = 69%, Cumulative CPU 827.74 sec
2013-09-17 23:59:16,046 Stage-1 map = 100%, reduce = 73%, Cumulative CPU 827.74 sec
2013-09-17 23:59:17,052 Stage-1 map = 100%, reduce = 75%, Cumulative CPU 827.74 sec
2013-09-17 23:59:18,059 Stage-1 map = 100%, reduce = 79%, Cumulative CPU 827.74 sec
2013-09-17 23:59:19,065 Stage-1 map = 100%, reduce = 83%, Cumulative CPU 827.74 sec
2013-09-17 23:59:20,072 Stage-1 map = 100%, reduce = 85%, Cumulative CPU 827.74 sec
2013-09-17 23:59:21,077 Stage-1 map = 100%, reduce = 89%, Cumulative CPU 827.74 sec
2013-09-17 23:59:22,114 Stage-1 map = 100%, reduce = 93%, Cumulative CPU 827.74 sec
2013-09-17 23:59:23,122 Stage-1 map = 100%, reduce = 95%, Cumulative CPU 871.5 sec
2013-09-17 23:59:24,133 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 1068.13 sec
2013-09-17 23:59:25,139 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 1068.13 sec
2013-09-17 23:59:26,143 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 1068.13 sec
MapReduce Total cumulative CPU time: 17 minutes 48 seconds 130 msec
Ended Job = job_201309172235_0079
Launching Job 2 out of 2
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0080
Hadoop job information for Stage-2: number of mappers: 5; number of reducers: 1
2013-09-17 23:59:29,594 Stage-2 map = 0%, reduce = 0%
2013-09-17 23:59:39,626 Stage-2 map = 6%, reduce = 0%
2013-09-17 23:59:42,636 Stage-2 map = 28%, reduce = 0%
2013-09-17 23:59:45,462 Stage-2 map = 39%, reduce = 0%
2013-09-17 23:59:46,467 Stage-2 map = 44%, reduce = 0%
2013-09-17 23:59:47,945 Stage-2 map = 52%, reduce = 0%, Cumulative CPU 23.32 sec
2013-09-17 23:59:48,950 Stage-2 map = 58%, reduce = 0%, Cumulative CPU 23.32 sec
2013-09-17 23:59:49,955 Stage-2 map = 58%, reduce = 0%, Cumulative CPU 23.32 sec
2013-09-17 23:59:50,960 Stage-2 map = 58%, reduce = 0%, Cumulative CPU 23.32 sec
2013-09-17 23:59:51,965 Stage-2 map = 63%, reduce = 0%, Cumulative CPU 23.32 sec
2013-09-17 23:59:52,970 Stage-2 map = 63%, reduce = 0%, Cumulative CPU 23.32 sec
2013-09-17 23:59:53,976 Stage-2 map = 63%, reduce = 0%, Cumulative CPU 23.32 sec
2013-09-17 23:59:54,981 Stage-2 map = 78%, reduce = 7%, Cumulative CPU 23.32 sec
2013-09-17 23:59:55,986 Stage-2 map = 78%, reduce = 7%, Cumulative CPU 23.32 sec
2013-09-17 23:59:56,992 Stage-2 map = 78%, reduce = 7%, Cumulative CPU 23.32 sec
2013-09-17 23:59:57,997 Stage-2 map = 84%, reduce = 7%, Cumulative CPU 23.32 sec
2013-09-17 23:59:59,002 Stage-2 map = 84%, reduce = 7%, Cumulative CPU 23.32 sec
2013-09-18 00:00:00,007 Stage-2 map = 84%, reduce = 7%, Cumulative CPU 23.32 sec
2013-09-18 00:00:01,028 Stage-2 map = 84%, reduce = 7%, Cumulative CPU 23.32 sec
2013-09-18 00:00:02,033 Stage-2 map = 84%, reduce = 7%, Cumulative CPU 23.32 sec
2013-09-18 00:00:03,039 Stage-2 map = 84%, reduce = 7%, Cumulative CPU 23.32 sec
2013-09-18 00:00:04,043 Stage-2 map = 87%, reduce = 7%, Cumulative CPU 66.8 sec
2013-09-18 00:00:05,049 Stage-2 map = 91%, reduce = 7%, Cumulative CPU 111.39 sec
2013-09-18 00:00:06,054 Stage-2 map = 91%, reduce = 7%, Cumulative CPU 111.39 sec
2013-09-18 00:00:07,058 Stage-2 map = 95%, reduce = 7%, Cumulative CPU 158.23 sec
2013-09-18 00:00:08,064 Stage-2 map = 95%, reduce = 7%, Cumulative CPU 158.23 sec
2013-09-18 00:00:09,069 Stage-2 map = 100%, reduce = 7%, Cumulative CPU 206.95 sec
2013-09-18 00:00:10,074 Stage-2 map = 100%, reduce = 7%, Cumulative CPU 206.95 sec
2013-09-18 00:00:11,079 Stage-2 map = 100%, reduce = 7%, Cumulative CPU 206.95 sec
2013-09-18 00:00:12,088 Stage-2 map = 100%, reduce = 7%, Cumulative CPU 206.95 sec
2013-09-18 00:00:13,092 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 206.95 sec
2013-09-18 00:00:14,097 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 206.95 sec
2013-09-18 00:00:15,102 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 206.95 sec
2013-09-18 00:00:16,106 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 206.95 sec
2013-09-18 00:00:17,110 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 206.95 sec
2013-09-18 00:00:18,114 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 206.95 sec
2013-09-18 00:00:19,118 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 206.95 sec
2013-09-18 00:00:20,123 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 206.95 sec
2013-09-18 00:00:21,127 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 206.95 sec
2013-09-18 00:00:22,131 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 206.95 sec
2013-09-18 00:00:23,135 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 206.95 sec
2013-09-18 00:00:24,139 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 206.95 sec
2013-09-18 00:00:25,144 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 206.95 sec
2013-09-18 00:00:26,148 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 206.95 sec
2013-09-18 00:00:27,152 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 206.95 sec
2013-09-18 00:00:28,157 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 226.88 sec
2013-09-18 00:00:29,161 Stage-2 map = 100%, reduce = 77%, Cumulative CPU 226.88 sec
2013-09-18 00:00:30,166 Stage-2 map = 100%, reduce = 77%, Cumulative CPU 226.88 sec
2013-09-18 00:00:31,170 Stage-2 map = 100%, reduce = 77%, Cumulative CPU 226.88 sec
2013-09-18 00:00:32,175 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 233.37 sec
2013-09-18 00:00:33,180 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 233.37 sec
MapReduce Total cumulative CPU time: 3 minutes 53 seconds 370 msec
Ended Job = job_201309172235_0080
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 11 Cumulative CPU: 1068.13 sec HDFS Read: 840872065 HDFS Write: 1196040653 SUCCESS
Job 1: Map: 5 Reduce: 1 Cumulative CPU: 233.37 sec HDFS Read: 1196066509 HDFS Write: 261 SUCCESS
Total MapReduce CPU Time Spent: 21 minutes 41 seconds 500 msec
OK
Time taken: 141.357 seconds, Fetched: 10 row(s)
hive> quit;
times: 2
query: SELECT UserID, SearchPhrase, count(*) AS c FROM hits_100m GROUP BY UserID, SearchPhrase ORDER BY c DESC LIMIT 10;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_8249@mturlrep13_201309180000_1745995125.txt
hive> SELECT UserID, SearchPhrase, count(*) AS c FROM hits_100m GROUP BY UserID, SearchPhrase ORDER BY c DESC LIMIT 10;;
Total MapReduce jobs = 2
Launching Job 1 out of 2
Number of reduce tasks not specified. Estimated from input data size: 11
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0081
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 11
2013-09-18 00:00:43,931 Stage-1 map = 0%, reduce = 0%
2013-09-18 00:00:56,183 Stage-1 map = 6%, reduce = 0%
2013-09-18 00:00:57,190 Stage-1 map = 7%, reduce = 0%
2013-09-18 00:00:58,202 Stage-1 map = 11%, reduce = 0%
2013-09-18 00:00:59,212 Stage-1 map = 29%, reduce = 0%
2013-09-18 00:01:00,225 Stage-1 map = 34%, reduce = 0%
2013-09-18 00:01:01,231 Stage-1 map = 37%, reduce = 0%
2013-09-18 00:01:02,237 Stage-1 map = 45%, reduce = 0%
2013-09-18 00:01:03,242 Stage-1 map = 58%, reduce = 0%
2013-09-18 00:01:04,254 Stage-1 map = 62%, reduce = 0%
2013-09-18 00:01:05,260 Stage-1 map = 66%, reduce = 0%
2013-09-18 00:01:06,286 Stage-1 map = 72%, reduce = 0%, Cumulative CPU 14.75 sec
2013-09-18 00:01:07,294 Stage-1 map = 74%, reduce = 0%, Cumulative CPU 30.84 sec
2013-09-18 00:01:08,306 Stage-1 map = 76%, reduce = 0%, Cumulative CPU 81.13 sec
2013-09-18 00:01:09,336 Stage-1 map = 79%, reduce = 0%, Cumulative CPU 133.53 sec
2013-09-18 00:01:10,343 Stage-1 map = 81%, reduce = 0%, Cumulative CPU 152.09 sec
2013-09-18 00:01:11,353 Stage-1 map = 82%, reduce = 0%, Cumulative CPU 212.05 sec
2013-09-18 00:01:12,359 Stage-1 map = 84%, reduce = 0%, Cumulative CPU 251.83 sec
2013-09-18 00:01:13,366 Stage-1 map = 87%, reduce = 0%, Cumulative CPU 359.32 sec
2013-09-18 00:01:14,373 Stage-1 map = 89%, reduce = 0%, Cumulative CPU 452.13 sec
2013-09-18 00:01:15,379 Stage-1 map = 93%, reduce = 0%, Cumulative CPU 547.58 sec
2013-09-18 00:01:16,387 Stage-1 map = 95%, reduce = 0%, Cumulative CPU 547.58 sec
2013-09-18 00:01:17,393 Stage-1 map = 97%, reduce = 2%, Cumulative CPU 603.6 sec
2013-09-18 00:01:18,400 Stage-1 map = 98%, reduce = 6%, Cumulative CPU 715.99 sec
2013-09-18 00:01:19,407 Stage-1 map = 99%, reduce = 18%, Cumulative CPU 744.69 sec
2013-09-18 00:01:20,413 Stage-1 map = 100%, reduce = 23%, Cumulative CPU 837.42 sec
2013-09-18 00:01:21,419 Stage-1 map = 100%, reduce = 25%, Cumulative CPU 837.42 sec
2013-09-18 00:01:22,426 Stage-1 map = 100%, reduce = 27%, Cumulative CPU 837.42 sec
2013-09-18 00:01:23,432 Stage-1 map = 100%, reduce = 29%, Cumulative CPU 837.42 sec
2013-09-18 00:01:24,439 Stage-1 map = 100%, reduce = 30%, Cumulative CPU 837.42 sec
2013-09-18 00:01:25,446 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 837.42 sec
2013-09-18 00:01:26,452 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 837.42 sec
2013-09-18 00:01:27,460 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 837.42 sec
2013-09-18 00:01:28,467 Stage-1 map = 100%, reduce = 36%, Cumulative CPU 890.75 sec
2013-09-18 00:01:29,474 Stage-1 map = 100%, reduce = 39%, Cumulative CPU 890.75 sec
2013-09-18 00:01:30,489 Stage-1 map = 100%, reduce = 53%, Cumulative CPU 890.75 sec
2013-09-18 00:01:31,496 Stage-1 map = 100%, reduce = 68%, Cumulative CPU 890.75 sec
2013-09-18 00:01:32,503 Stage-1 map = 100%, reduce = 72%, Cumulative CPU 890.75 sec
2013-09-18 00:01:33,509 Stage-1 map = 100%, reduce = 75%, Cumulative CPU 890.75 sec
2013-09-18 00:01:34,515 Stage-1 map = 100%, reduce = 80%, Cumulative CPU 890.75 sec
2013-09-18 00:01:35,522 Stage-1 map = 100%, reduce = 82%, Cumulative CPU 890.75 sec
2013-09-18 00:01:36,528 Stage-1 map = 100%, reduce = 85%, Cumulative CPU 890.75 sec
2013-09-18 00:01:37,534 Stage-1 map = 100%, reduce = 89%, Cumulative CPU 890.75 sec
2013-09-18 00:01:38,541 Stage-1 map = 100%, reduce = 92%, Cumulative CPU 890.75 sec
2013-09-18 00:01:39,555 Stage-1 map = 100%, reduce = 97%, Cumulative CPU 998.36 sec
2013-09-18 00:01:40,562 Stage-1 map = 100%, reduce = 97%, Cumulative CPU 998.36 sec
2013-09-18 00:01:41,568 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1035.02 sec
2013-09-18 00:01:42,574 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 1070.49 sec
2013-09-18 00:01:43,580 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 1070.49 sec
MapReduce Total cumulative CPU time: 17 minutes 50 seconds 490 msec
Ended Job = job_201309172235_0081
Launching Job 2 out of 2
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0082
Hadoop job information for Stage-2: number of mappers: 5; number of reducers: 1
2013-09-18 00:01:47,035 Stage-2 map = 0%, reduce = 0%
2013-09-18 00:01:57,066 Stage-2 map = 6%, reduce = 0%
2013-09-18 00:02:00,076 Stage-2 map = 28%, reduce = 0%
2013-09-18 00:02:05,092 Stage-2 map = 49%, reduce = 0%, Cumulative CPU 22.46 sec
2013-09-18 00:02:06,098 Stage-2 map = 49%, reduce = 0%, Cumulative CPU 22.46 sec
2013-09-18 00:02:07,104 Stage-2 map = 49%, reduce = 0%, Cumulative CPU 22.46 sec
2013-09-18 00:02:10,085 Stage-2 map = 52%, reduce = 0%, Cumulative CPU 22.46 sec
2013-09-18 00:02:11,090 Stage-2 map = 52%, reduce = 0%, Cumulative CPU 22.46 sec
2013-09-18 00:02:12,095 Stage-2 map = 52%, reduce = 0%, Cumulative CPU 22.46 sec
2013-09-18 00:02:13,100 Stage-2 map = 58%, reduce = 7%, Cumulative CPU 22.46 sec
2013-09-18 00:02:14,106 Stage-2 map = 58%, reduce = 7%, Cumulative CPU 22.46 sec
2013-09-18 00:02:15,111 Stage-2 map = 58%, reduce = 7%, Cumulative CPU 22.46 sec
2013-09-18 00:02:16,117 Stage-2 map = 68%, reduce = 7%, Cumulative CPU 22.46 sec
2013-09-18 00:02:17,122 Stage-2 map = 68%, reduce = 7%, Cumulative CPU 22.46 sec
2013-09-18 00:02:18,128 Stage-2 map = 68%, reduce = 7%, Cumulative CPU 22.46 sec
2013-09-18 00:02:19,133 Stage-2 map = 84%, reduce = 7%, Cumulative CPU 22.46 sec
2013-09-18 00:02:20,166 Stage-2 map = 84%, reduce = 7%, Cumulative CPU 22.46 sec
2013-09-18 00:02:21,171 Stage-2 map = 84%, reduce = 7%, Cumulative CPU 22.46 sec
2013-09-18 00:02:22,176 Stage-2 map = 84%, reduce = 7%, Cumulative CPU 22.46 sec
2013-09-18 00:02:23,273 Stage-2 map = 84%, reduce = 7%, Cumulative CPU 22.46 sec
2013-09-18 00:02:24,279 Stage-2 map = 84%, reduce = 7%, Cumulative CPU 22.46 sec
2013-09-18 00:02:25,284 Stage-2 map = 84%, reduce = 7%, Cumulative CPU 22.46 sec
2013-09-18 00:02:26,289 Stage-2 map = 91%, reduce = 7%, Cumulative CPU 110.5 sec
2013-09-18 00:02:27,294 Stage-2 map = 91%, reduce = 7%, Cumulative CPU 110.5 sec
2013-09-18 00:02:28,412 Stage-2 map = 91%, reduce = 7%, Cumulative CPU 203.81 sec
2013-09-18 00:02:29,416 Stage-2 map = 100%, reduce = 7%, Cumulative CPU 206.44 sec
2013-09-18 00:02:30,420 Stage-2 map = 100%, reduce = 7%, Cumulative CPU 206.44 sec
2013-09-18 00:02:31,425 Stage-2 map = 100%, reduce = 20%, Cumulative CPU 206.44 sec
2013-09-18 00:02:32,429 Stage-2 map = 100%, reduce = 20%, Cumulative CPU 206.44 sec
2013-09-18 00:02:33,989 Stage-2 map = 100%, reduce = 20%, Cumulative CPU 206.44 sec
2013-09-18 00:02:34,994 Stage-2 map = 100%, reduce = 20%, Cumulative CPU 206.44 sec
2013-09-18 00:02:35,999 Stage-2 map = 100%, reduce = 20%, Cumulative CPU 206.44 sec
2013-09-18 00:02:37,003 Stage-2 map = 100%, reduce = 20%, Cumulative CPU 206.44 sec
2013-09-18 00:02:38,007 Stage-2 map = 100%, reduce = 20%, Cumulative CPU 206.44 sec
2013-09-18 00:02:39,011 Stage-2 map = 100%, reduce = 20%, Cumulative CPU 206.44 sec
2013-09-18 00:02:40,015 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 206.44 sec
2013-09-18 00:02:41,020 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 206.44 sec
2013-09-18 00:02:42,023 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 206.44 sec
2013-09-18 00:02:43,027 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 206.44 sec
2013-09-18 00:02:44,031 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 206.44 sec
2013-09-18 00:02:45,036 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 206.44 sec
2013-09-18 00:02:46,040 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 206.44 sec
2013-09-18 00:02:47,045 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 206.44 sec
2013-09-18 00:02:48,049 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 206.44 sec
2013-09-18 00:02:49,054 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 206.44 sec
2013-09-18 00:02:50,058 Stage-2 map = 100%, reduce = 69%, Cumulative CPU 206.44 sec
2013-09-18 00:02:51,063 Stage-2 map = 100%, reduce = 69%, Cumulative CPU 206.44 sec
2013-09-18 00:02:52,067 Stage-2 map = 100%, reduce = 69%, Cumulative CPU 206.44 sec
2013-09-18 00:02:53,071 Stage-2 map = 100%, reduce = 77%, Cumulative CPU 206.44 sec
2013-09-18 00:02:54,395 Stage-2 map = 100%, reduce = 77%, Cumulative CPU 206.44 sec
2013-09-18 00:02:55,399 Stage-2 map = 100%, reduce = 77%, Cumulative CPU 206.44 sec
2013-09-18 00:02:56,403 Stage-2 map = 100%, reduce = 77%, Cumulative CPU 206.44 sec
2013-09-18 00:02:57,408 Stage-2 map = 100%, reduce = 77%, Cumulative CPU 206.44 sec
2013-09-18 00:03:01,104 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 232.56 sec
2013-09-18 00:03:02,108 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 232.56 sec
MapReduce Total cumulative CPU time: 3 minutes 52 seconds 560 msec
Ended Job = job_201309172235_0082
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 11 Cumulative CPU: 1070.49 sec HDFS Read: 840872065 HDFS Write: 1196040653 SUCCESS
Job 1: Map: 5 Reduce: 1 Cumulative CPU: 232.56 sec HDFS Read: 1196066509 HDFS Write: 261 SUCCESS
Total MapReduce CPU Time Spent: 21 minutes 43 seconds 50 msec
OK
Time taken: 147.059 seconds, Fetched: 10 row(s)
hive> quit;
times: 3
query: SELECT UserID, SearchPhrase, count(*) AS c FROM hits_100m GROUP BY UserID, SearchPhrase ORDER BY c DESC LIMIT 10;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_15858@mturlrep13_201309180003_1278146290.txt
hive> SELECT UserID, SearchPhrase, count(*) AS c FROM hits_100m GROUP BY UserID, SearchPhrase ORDER BY c DESC LIMIT 10;;
Total MapReduce jobs = 2
Launching Job 1 out of 2
Number of reduce tasks not specified. Estimated from input data size: 11
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0083
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 11
2013-09-18 00:03:11,727 Stage-1 map = 0%, reduce = 0%
2013-09-18 00:03:23,895 Stage-1 map = 2%, reduce = 0%
2013-09-18 00:03:24,924 Stage-1 map = 5%, reduce = 0%
2013-09-18 00:03:25,943 Stage-1 map = 6%, reduce = 0%
2013-09-18 00:03:26,979 Stage-1 map = 13%, reduce = 0%
2013-09-18 00:03:28,028 Stage-1 map = 34%, reduce = 0%, Cumulative CPU 341.29 sec
2013-09-18 00:03:29,044 Stage-1 map = 34%, reduce = 0%, Cumulative CPU 341.29 sec
2013-09-18 00:03:30,053 Stage-1 map = 36%, reduce = 0%, Cumulative CPU 341.29 sec
2013-09-18 00:03:31,062 Stage-1 map = 54%, reduce = 0%, Cumulative CPU 341.29 sec
2013-09-18 00:03:32,074 Stage-1 map = 59%, reduce = 0%, Cumulative CPU 341.29 sec
2013-09-18 00:03:33,090 Stage-1 map = 60%, reduce = 0%, Cumulative CPU 346.1 sec
2013-09-18 00:03:34,112 Stage-1 map = 67%, reduce = 0%, Cumulative CPU 346.1 sec
2013-09-18 00:03:35,122 Stage-1 map = 73%, reduce = 0%, Cumulative CPU 346.1 sec
2013-09-18 00:03:36,133 Stage-1 map = 73%, reduce = 0%, Cumulative CPU 346.1 sec
2013-09-18 00:03:37,145 Stage-1 map = 79%, reduce = 0%, Cumulative CPU 394.29 sec
2013-09-18 00:03:38,153 Stage-1 map = 79%, reduce = 0%, Cumulative CPU 402.8 sec
2013-09-18 00:03:39,160 Stage-1 map = 80%, reduce = 0%, Cumulative CPU 413.32 sec
2013-09-18 00:03:40,175 Stage-1 map = 82%, reduce = 0%, Cumulative CPU 435.9 sec
2013-09-18 00:03:41,200 Stage-1 map = 87%, reduce = 0%, Cumulative CPU 505.2 sec
2013-09-18 00:03:42,209 Stage-1 map = 88%, reduce = 0%, Cumulative CPU 544.32 sec
2013-09-18 00:03:43,217 Stage-1 map = 92%, reduce = 0%, Cumulative CPU 600.85 sec
2013-09-18 00:03:44,223 Stage-1 map = 95%, reduce = 0%, Cumulative CPU 629.83 sec
2013-09-18 00:03:45,230 Stage-1 map = 95%, reduce = 0%, Cumulative CPU 645.26 sec
2013-09-18 00:03:46,236 Stage-1 map = 96%, reduce = 0%, Cumulative CPU 682.7 sec
2013-09-18 00:03:47,243 Stage-1 map = 98%, reduce = 3%, Cumulative CPU 758.48 sec
2013-09-18 00:03:48,250 Stage-1 map = 99%, reduce = 17%, Cumulative CPU 799.32 sec
2013-09-18 00:03:49,257 Stage-1 map = 100%, reduce = 22%, Cumulative CPU 841.01 sec
2013-09-18 00:03:50,263 Stage-1 map = 100%, reduce = 24%, Cumulative CPU 841.01 sec
2013-09-18 00:03:51,269 Stage-1 map = 100%, reduce = 26%, Cumulative CPU 841.01 sec
2013-09-18 00:03:52,275 Stage-1 map = 100%, reduce = 30%, Cumulative CPU 841.01 sec
2013-09-18 00:03:53,281 Stage-1 map = 100%, reduce = 31%, Cumulative CPU 841.01 sec
2013-09-18 00:03:54,287 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 841.01 sec
2013-09-18 00:03:55,293 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 841.01 sec
2013-09-18 00:03:56,300 Stage-1 map = 100%, reduce = 36%, Cumulative CPU 841.01 sec
2013-09-18 00:03:57,306 Stage-1 map = 100%, reduce = 42%, Cumulative CPU 841.01 sec
2013-09-18 00:03:58,313 Stage-1 map = 100%, reduce = 49%, Cumulative CPU 841.01 sec
2013-09-18 00:03:59,319 Stage-1 map = 100%, reduce = 53%, Cumulative CPU 841.01 sec
2013-09-18 00:04:00,325 Stage-1 map = 100%, reduce = 61%, Cumulative CPU 841.01 sec
2013-09-18 00:04:01,331 Stage-1 map = 100%, reduce = 72%, Cumulative CPU 841.01 sec
2013-09-18 00:04:02,338 Stage-1 map = 100%, reduce = 74%, Cumulative CPU 841.01 sec
2013-09-18 00:04:03,344 Stage-1 map = 100%, reduce = 78%, Cumulative CPU 841.01 sec
2013-09-18 00:04:04,350 Stage-1 map = 100%, reduce = 81%, Cumulative CPU 841.01 sec
2013-09-18 00:04:05,357 Stage-1 map = 100%, reduce = 84%, Cumulative CPU 841.01 sec
2013-09-18 00:04:06,363 Stage-1 map = 100%, reduce = 88%, Cumulative CPU 841.01 sec
2013-09-18 00:04:07,382 Stage-1 map = 100%, reduce = 91%, Cumulative CPU 841.01 sec
2013-09-18 00:04:08,391 Stage-1 map = 100%, reduce = 95%, Cumulative CPU 924.61 sec
2013-09-18 00:04:09,396 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1053.16 sec
2013-09-18 00:04:10,402 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 1074.59 sec
2013-09-18 00:04:11,408 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 1074.59 sec
MapReduce Total cumulative CPU time: 17 minutes 54 seconds 590 msec
Ended Job = job_201309172235_0083
Launching Job 2 out of 2
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0084
Hadoop job information for Stage-2: number of mappers: 5; number of reducers: 1
2013-09-18 00:04:16,454 Stage-2 map = 0%, reduce = 0%
2013-09-18 00:04:27,489 Stage-2 map = 6%, reduce = 0%
2013-09-18 00:04:30,502 Stage-2 map = 28%, reduce = 0%, Cumulative CPU 71.89 sec
2013-09-18 00:04:31,507 Stage-2 map = 28%, reduce = 0%, Cumulative CPU 71.89 sec
2013-09-18 00:04:32,513 Stage-2 map = 28%, reduce = 0%, Cumulative CPU 71.89 sec
2013-09-18 00:04:33,518 Stage-2 map = 28%, reduce = 0%, Cumulative CPU 71.89 sec
2013-09-18 00:04:34,523 Stage-2 map = 41%, reduce = 0%, Cumulative CPU 71.89 sec
2013-09-18 00:04:35,529 Stage-2 map = 49%, reduce = 0%, Cumulative CPU 81.02 sec
2013-09-18 00:04:36,534 Stage-2 map = 49%, reduce = 0%, Cumulative CPU 81.02 sec
2013-09-18 00:04:37,539 Stage-2 map = 49%, reduce = 0%, Cumulative CPU 81.02 sec
2013-09-18 00:04:38,545 Stage-2 map = 58%, reduce = 0%, Cumulative CPU 81.02 sec
2013-09-18 00:04:39,550 Stage-2 map = 58%, reduce = 0%, Cumulative CPU 81.02 sec
2013-09-18 00:04:41,187 Stage-2 map = 68%, reduce = 0%, Cumulative CPU 81.02 sec
2013-09-18 00:04:42,265 Stage-2 map = 68%, reduce = 0%, Cumulative CPU 81.02 sec
2013-09-18 00:04:43,270 Stage-2 map = 68%, reduce = 7%, Cumulative CPU 81.02 sec
2013-09-18 00:04:44,275 Stage-2 map = 84%, reduce = 7%, Cumulative CPU 81.02 sec
2013-09-18 00:04:45,281 Stage-2 map = 84%, reduce = 7%, Cumulative CPU 81.02 sec
2013-09-18 00:04:46,286 Stage-2 map = 84%, reduce = 7%, Cumulative CPU 81.02 sec
2013-09-18 00:04:47,291 Stage-2 map = 84%, reduce = 7%, Cumulative CPU 81.02 sec
2013-09-18 00:04:48,302 Stage-2 map = 84%, reduce = 7%, Cumulative CPU 81.02 sec
2013-09-18 00:04:49,306 Stage-2 map = 84%, reduce = 7%, Cumulative CPU 81.02 sec
2013-09-18 00:04:50,311 Stage-2 map = 84%, reduce = 7%, Cumulative CPU 81.02 sec
2013-09-18 00:04:51,316 Stage-2 map = 87%, reduce = 7%, Cumulative CPU 108.7 sec
2013-09-18 00:04:52,321 Stage-2 map = 91%, reduce = 7%, Cumulative CPU 136.95 sec
2013-09-18 00:04:53,326 Stage-2 map = 91%, reduce = 7%, Cumulative CPU 136.95 sec
2013-09-18 00:04:54,331 Stage-2 map = 91%, reduce = 7%, Cumulative CPU 136.95 sec
2013-09-18 00:04:55,335 Stage-2 map = 95%, reduce = 7%, Cumulative CPU 169.96 sec
2013-09-18 00:04:56,340 Stage-2 map = 100%, reduce = 7%, Cumulative CPU 203.62 sec
2013-09-18 00:04:57,344 Stage-2 map = 100%, reduce = 7%, Cumulative CPU 203.62 sec
2013-09-18 00:04:58,348 Stage-2 map = 100%, reduce = 20%, Cumulative CPU 203.62 sec
2013-09-18 00:04:59,353 Stage-2 map = 100%, reduce = 20%, Cumulative CPU 203.62 sec
2013-09-18 00:05:00,358 Stage-2 map = 100%, reduce = 20%, Cumulative CPU 203.62 sec
2013-09-18 00:05:01,362 Stage-2 map = 100%, reduce = 20%, Cumulative CPU 203.62 sec
2013-09-18 00:05:02,367 Stage-2 map = 100%, reduce = 20%, Cumulative CPU 203.62 sec
2013-09-18 00:05:03,371 Stage-2 map = 100%, reduce = 20%, Cumulative CPU 203.62 sec
2013-09-18 00:05:04,376 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 203.62 sec
2013-09-18 00:05:05,380 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 203.62 sec
2013-09-18 00:05:06,384 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 203.62 sec
2013-09-18 00:05:07,389 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 203.62 sec
2013-09-18 00:05:08,393 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 203.62 sec
2013-09-18 00:05:09,397 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 203.62 sec
2013-09-18 00:05:10,402 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 203.62 sec
2013-09-18 00:05:11,406 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 203.62 sec
2013-09-18 00:05:12,410 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 203.62 sec
2013-09-18 00:05:13,415 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 203.62 sec
2013-09-18 00:05:14,419 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 203.62 sec
2013-09-18 00:05:15,422 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 203.62 sec
2013-09-18 00:05:16,426 Stage-2 map = 100%, reduce = 77%, Cumulative CPU 203.62 sec
2013-09-18 00:05:17,430 Stage-2 map = 100%, reduce = 77%, Cumulative CPU 203.62 sec
2013-09-18 00:05:18,433 Stage-2 map = 100%, reduce = 77%, Cumulative CPU 203.62 sec
2013-09-18 00:05:19,439 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 228.63 sec
2013-09-18 00:05:20,443 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 228.63 sec
2013-09-18 00:05:21,448 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 228.63 sec
2013-09-18 00:05:24,696 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 228.63 sec
MapReduce Total cumulative CPU time: 3 minutes 48 seconds 630 msec
Ended Job = job_201309172235_0084
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 11 Cumulative CPU: 1074.59 sec HDFS Read: 840872065 HDFS Write: 1196040653 SUCCESS
Job 1: Map: 5 Reduce: 1 Cumulative CPU: 228.63 sec HDFS Read: 1196066487 HDFS Write: 261 SUCCESS
Total MapReduce CPU Time Spent: 21 minutes 43 seconds 220 msec
OK
Time taken: 140.75 seconds, Fetched: 10 row(s)
hive> quit;
-- ещё более сложная агрегация.;
times: 1
query: SELECT UserID, SearchPhrase, count(*) AS c FROM hits_100m GROUP BY UserID, SearchPhrase LIMIT 10;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_24153@mturlrep13_201309180005_705271963.txt
hive> SELECT UserID, SearchPhrase, count(*) AS c FROM hits_100m GROUP BY UserID, SearchPhrase LIMIT 10;;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks not specified. Estimated from input data size: 11
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0085
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 11
2013-09-18 00:05:46,233 Stage-1 map = 0%, reduce = 0%
2013-09-18 00:05:56,352 Stage-1 map = 3%, reduce = 0%
2013-09-18 00:05:59,390 Stage-1 map = 10%, reduce = 0%
2013-09-18 00:06:00,398 Stage-1 map = 17%, reduce = 0%
2013-09-18 00:06:02,423 Stage-1 map = 21%, reduce = 0%
2013-09-18 00:06:03,429 Stage-1 map = 28%, reduce = 0%
2013-09-18 00:06:04,764 Stage-1 map = 32%, reduce = 0%
2013-09-18 00:06:05,770 Stage-1 map = 40%, reduce = 0%
2013-09-18 00:06:06,782 Stage-1 map = 46%, reduce = 0%
2013-09-18 00:06:07,789 Stage-1 map = 49%, reduce = 0%
2013-09-18 00:06:08,936 Stage-1 map = 57%, reduce = 0%
2013-09-18 00:06:09,962 Stage-1 map = 60%, reduce = 0%, Cumulative CPU 31.6 sec
2013-09-18 00:06:11,308 Stage-1 map = 67%, reduce = 0%, Cumulative CPU 48.37 sec
2013-09-18 00:06:12,316 Stage-1 map = 73%, reduce = 0%, Cumulative CPU 48.37 sec
2013-09-18 00:06:13,322 Stage-1 map = 77%, reduce = 0%, Cumulative CPU 70.19 sec
2013-09-18 00:06:14,328 Stage-1 map = 78%, reduce = 0%, Cumulative CPU 89.75 sec
2013-09-18 00:06:15,335 Stage-1 map = 80%, reduce = 0%, Cumulative CPU 106.46 sec
2013-09-18 00:06:16,340 Stage-1 map = 82%, reduce = 0%, Cumulative CPU 181.45 sec
2013-09-18 00:06:17,350 Stage-1 map = 84%, reduce = 0%, Cumulative CPU 234.9 sec
2013-09-18 00:06:18,357 Stage-1 map = 85%, reduce = 0%, Cumulative CPU 234.9 sec
2013-09-18 00:06:19,363 Stage-1 map = 90%, reduce = 0%, Cumulative CPU 376.71 sec
2013-09-18 00:06:20,369 Stage-1 map = 92%, reduce = 0%, Cumulative CPU 461.07 sec
2013-09-18 00:06:21,375 Stage-1 map = 94%, reduce = 0%, Cumulative CPU 534.4 sec
2013-09-18 00:06:22,382 Stage-1 map = 96%, reduce = 1%, Cumulative CPU 584.98 sec
2013-09-18 00:06:23,388 Stage-1 map = 97%, reduce = 5%, Cumulative CPU 641.71 sec
2013-09-18 00:06:24,393 Stage-1 map = 99%, reduce = 5%, Cumulative CPU 730.93 sec
2013-09-18 00:06:25,398 Stage-1 map = 99%, reduce = 5%, Cumulative CPU 792.13 sec
2013-09-18 00:06:26,404 Stage-1 map = 100%, reduce = 16%, Cumulative CPU 822.69 sec
2013-09-18 00:06:27,410 Stage-1 map = 100%, reduce = 28%, Cumulative CPU 822.69 sec
2013-09-18 00:06:28,416 Stage-1 map = 100%, reduce = 28%, Cumulative CPU 822.69 sec
2013-09-18 00:06:29,422 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 853.5 sec
2013-09-18 00:06:30,427 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 853.5 sec
2013-09-18 00:06:31,434 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 853.5 sec
2013-09-18 00:06:32,440 Stage-1 map = 100%, reduce = 36%, Cumulative CPU 853.5 sec
2013-09-18 00:06:33,447 Stage-1 map = 100%, reduce = 67%, Cumulative CPU 890.94 sec
2013-09-18 00:06:34,456 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 928.82 sec
2013-09-18 00:06:35,461 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 928.82 sec
2013-09-18 00:06:36,466 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 928.82 sec
MapReduce Total cumulative CPU time: 15 minutes 28 seconds 820 msec
Ended Job = job_201309172235_0085
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 11 Cumulative CPU: 928.82 sec HDFS Read: 840872065 HDFS Write: 5012 SUCCESS
Total MapReduce CPU Time Spent: 15 minutes 28 seconds 820 msec
OK
Time taken: 60.884 seconds, Fetched: 10 row(s)
hive> quit;
times: 2
query: SELECT UserID, SearchPhrase, count(*) AS c FROM hits_100m GROUP BY UserID, SearchPhrase LIMIT 10;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_30696@mturlrep13_201309180006_357154034.txt
hive> SELECT UserID, SearchPhrase, count(*) AS c FROM hits_100m GROUP BY UserID, SearchPhrase LIMIT 10;;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks not specified. Estimated from input data size: 11
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0086
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 11
2013-09-18 00:06:47,336 Stage-1 map = 0%, reduce = 0%
2013-09-18 00:06:58,492 Stage-1 map = 1%, reduce = 0%
2013-09-18 00:06:59,507 Stage-1 map = 5%, reduce = 0%
2013-09-18 00:07:00,512 Stage-1 map = 8%, reduce = 0%
2013-09-18 00:07:01,519 Stage-1 map = 11%, reduce = 0%
2013-09-18 00:07:02,525 Stage-1 map = 30%, reduce = 0%
2013-09-18 00:07:03,531 Stage-1 map = 33%, reduce = 0%
2013-09-18 00:07:04,537 Stage-1 map = 36%, reduce = 0%
2013-09-18 00:07:05,542 Stage-1 map = 51%, reduce = 0%
2013-09-18 00:07:06,547 Stage-1 map = 56%, reduce = 0%
2013-09-18 00:07:07,553 Stage-1 map = 60%, reduce = 0%
2013-09-18 00:07:08,565 Stage-1 map = 66%, reduce = 0%
2013-09-18 00:07:09,577 Stage-1 map = 70%, reduce = 0%, Cumulative CPU 14.92 sec
2013-09-18 00:07:10,584 Stage-1 map = 73%, reduce = 0%, Cumulative CPU 46.57 sec
2013-09-18 00:07:11,590 Stage-1 map = 76%, reduce = 0%, Cumulative CPU 80.51 sec
2013-09-18 00:07:12,596 Stage-1 map = 79%, reduce = 0%, Cumulative CPU 131.92 sec
2013-09-18 00:07:13,713 Stage-1 map = 80%, reduce = 0%, Cumulative CPU 131.92 sec
2013-09-18 00:07:14,736 Stage-1 map = 81%, reduce = 0%, Cumulative CPU 152.9 sec
2013-09-18 00:07:15,752 Stage-1 map = 84%, reduce = 0%, Cumulative CPU 254.96 sec
2013-09-18 00:07:16,766 Stage-1 map = 87%, reduce = 0%, Cumulative CPU 338.32 sec
2013-09-18 00:07:17,773 Stage-1 map = 89%, reduce = 0%, Cumulative CPU 429.15 sec
2013-09-18 00:07:18,778 Stage-1 map = 93%, reduce = 0%, Cumulative CPU 523.97 sec
2013-09-18 00:07:19,784 Stage-1 map = 95%, reduce = 0%, Cumulative CPU 548.71 sec
2013-09-18 00:07:20,791 Stage-1 map = 96%, reduce = 0%, Cumulative CPU 604.14 sec
2013-09-18 00:07:21,798 Stage-1 map = 97%, reduce = 2%, Cumulative CPU 660.44 sec
2013-09-18 00:07:22,804 Stage-1 map = 99%, reduce = 13%, Cumulative CPU 747.55 sec
2013-09-18 00:07:23,810 Stage-1 map = 100%, reduce = 18%, Cumulative CPU 839.86 sec
2013-09-18 00:07:24,816 Stage-1 map = 100%, reduce = 21%, Cumulative CPU 839.86 sec
2013-09-18 00:07:25,822 Stage-1 map = 100%, reduce = 21%, Cumulative CPU 839.86 sec
2013-09-18 00:07:26,829 Stage-1 map = 100%, reduce = 25%, Cumulative CPU 839.86 sec
2013-09-18 00:07:27,835 Stage-1 map = 100%, reduce = 30%, Cumulative CPU 839.86 sec
2013-09-18 00:07:28,841 Stage-1 map = 100%, reduce = 32%, Cumulative CPU 839.86 sec
2013-09-18 00:07:29,847 Stage-1 map = 100%, reduce = 32%, Cumulative CPU 871.07 sec
2013-09-18 00:07:30,853 Stage-1 map = 100%, reduce = 32%, Cumulative CPU 871.07 sec
2013-09-18 00:07:31,859 Stage-1 map = 100%, reduce = 36%, Cumulative CPU 871.07 sec
2013-09-18 00:07:32,868 Stage-1 map = 100%, reduce = 61%, Cumulative CPU 898.81 sec
2013-09-18 00:07:33,873 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 949.24 sec
2013-09-18 00:07:34,879 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 949.24 sec
2013-09-18 00:07:35,885 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 949.24 sec
MapReduce Total cumulative CPU time: 15 minutes 49 seconds 240 msec
Ended Job = job_201309172235_0086
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 11 Cumulative CPU: 949.24 sec HDFS Read: 840872065 HDFS Write: 5012 SUCCESS
Total MapReduce CPU Time Spent: 15 minutes 49 seconds 240 msec
OK
Time taken: 57.376 seconds, Fetched: 10 row(s)
hive> quit;
times: 3
query: SELECT UserID, SearchPhrase, count(*) AS c FROM hits_100m GROUP BY UserID, SearchPhrase LIMIT 10;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_4908@mturlrep13_201309180007_865221313.txt
hive> SELECT UserID, SearchPhrase, count(*) AS c FROM hits_100m GROUP BY UserID, SearchPhrase LIMIT 10;;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks not specified. Estimated from input data size: 11
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0087
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 11
2013-09-18 00:07:46,553 Stage-1 map = 0%, reduce = 0%
2013-09-18 00:07:56,648 Stage-1 map = 1%, reduce = 0%
2013-09-18 00:07:58,657 Stage-1 map = 5%, reduce = 0%
2013-09-18 00:07:59,663 Stage-1 map = 7%, reduce = 0%
2013-09-18 00:08:00,669 Stage-1 map = 12%, reduce = 0%
2013-09-18 00:08:01,675 Stage-1 map = 29%, reduce = 0%
2013-09-18 00:08:02,699 Stage-1 map = 32%, reduce = 0%
2013-09-18 00:08:03,714 Stage-1 map = 37%, reduce = 0%
2013-09-18 00:08:04,734 Stage-1 map = 46%, reduce = 0%
2013-09-18 00:08:05,741 Stage-1 map = 55%, reduce = 0%
2013-09-18 00:08:06,749 Stage-1 map = 61%, reduce = 0%
2013-09-18 00:08:07,780 Stage-1 map = 67%, reduce = 0%, Cumulative CPU 14.11 sec
2013-09-18 00:08:08,868 Stage-1 map = 70%, reduce = 0%, Cumulative CPU 14.11 sec
2013-09-18 00:08:09,882 Stage-1 map = 72%, reduce = 0%, Cumulative CPU 14.11 sec
2013-09-18 00:08:10,889 Stage-1 map = 77%, reduce = 0%, Cumulative CPU 64.17 sec
2013-09-18 00:08:11,894 Stage-1 map = 79%, reduce = 0%, Cumulative CPU 133.33 sec
2013-09-18 00:08:12,901 Stage-1 map = 79%, reduce = 0%, Cumulative CPU 152.87 sec
2013-09-18 00:08:13,906 Stage-1 map = 80%, reduce = 0%, Cumulative CPU 152.87 sec
2013-09-18 00:08:14,926 Stage-1 map = 86%, reduce = 0%, Cumulative CPU 318.88 sec
2013-09-18 00:08:15,942 Stage-1 map = 87%, reduce = 0%, Cumulative CPU 382.36 sec
2013-09-18 00:08:16,947 Stage-1 map = 91%, reduce = 0%, Cumulative CPU 498.71 sec
2013-09-18 00:08:17,953 Stage-1 map = 94%, reduce = 0%, Cumulative CPU 522.53 sec
2013-09-18 00:08:18,958 Stage-1 map = 94%, reduce = 0%, Cumulative CPU 546.88 sec
2013-09-18 00:08:19,985 Stage-1 map = 96%, reduce = 0%, Cumulative CPU 603.23 sec
2013-09-18 00:08:20,991 Stage-1 map = 97%, reduce = 2%, Cumulative CPU 630.86 sec
2013-09-18 00:08:21,996 Stage-1 map = 99%, reduce = 13%, Cumulative CPU 777.69 sec
2013-09-18 00:08:23,002 Stage-1 map = 100%, reduce = 21%, Cumulative CPU 839.76 sec
2013-09-18 00:08:24,008 Stage-1 map = 100%, reduce = 24%, Cumulative CPU 839.76 sec
2013-09-18 00:08:25,013 Stage-1 map = 100%, reduce = 27%, Cumulative CPU 839.76 sec
2013-09-18 00:08:26,019 Stage-1 map = 100%, reduce = 27%, Cumulative CPU 839.76 sec
2013-09-18 00:08:27,026 Stage-1 map = 100%, reduce = 28%, Cumulative CPU 839.76 sec
2013-09-18 00:08:28,032 Stage-1 map = 100%, reduce = 31%, Cumulative CPU 839.76 sec
2013-09-18 00:08:29,038 Stage-1 map = 100%, reduce = 31%, Cumulative CPU 839.76 sec
2013-09-18 00:08:30,044 Stage-1 map = 100%, reduce = 31%, Cumulative CPU 872.39 sec
2013-09-18 00:08:31,050 Stage-1 map = 100%, reduce = 38%, Cumulative CPU 872.39 sec
2013-09-18 00:08:32,059 Stage-1 map = 100%, reduce = 61%, Cumulative CPU 881.87 sec
2013-09-18 00:08:33,065 Stage-1 map = 100%, reduce = 88%, Cumulative CPU 922.01 sec
2013-09-18 00:08:34,070 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 945.45 sec
2013-09-18 00:08:35,076 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 945.45 sec
MapReduce Total cumulative CPU time: 15 minutes 45 seconds 450 msec
Ended Job = job_201309172235_0087
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 11 Cumulative CPU: 945.45 sec HDFS Read: 840872065 HDFS Write: 5012 SUCCESS
Total MapReduce CPU Time Spent: 15 minutes 45 seconds 450 msec
OK
Time taken: 57.367 seconds, Fetched: 10 row(s)
hive> quit;
-- то же самое, но без сортировки.;
times: 1
query: SELECT UserID, minute(EventTime), SearchPhrase, count(*) AS c FROM hits_100m GROUP BY UserID, minute(EventTime), SearchPhrase ORDER BY c DESC LIMIT 10;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_11923@mturlrep13_201309180008_1140241874.txt
hive> SELECT UserID, minute(EventTime), SearchPhrase, count(*) AS c FROM hits_100m GROUP BY UserID, minute(EventTime), SearchPhrase ORDER BY c DESC LIMIT 10;;
Total MapReduce jobs = 2
Launching Job 1 out of 2
Number of reduce tasks not specified. Estimated from input data size: 11
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0088
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 11
2013-09-18 00:09:03,020 Stage-1 map = 0%, reduce = 0%
2013-09-18 00:09:16,111 Stage-1 map = 1%, reduce = 0%
2013-09-18 00:09:17,117 Stage-1 map = 2%, reduce = 0%
2013-09-18 00:09:19,132 Stage-1 map = 3%, reduce = 0%
2013-09-18 00:09:20,138 Stage-1 map = 5%, reduce = 0%
2013-09-18 00:09:21,154 Stage-1 map = 8%, reduce = 0%
2013-09-18 00:09:22,176 Stage-1 map = 11%, reduce = 0%
2013-09-18 00:09:23,188 Stage-1 map = 12%, reduce = 0%
2013-09-18 00:09:24,193 Stage-1 map = 15%, reduce = 0%
2013-09-18 00:09:25,203 Stage-1 map = 22%, reduce = 0%
2013-09-18 00:09:26,210 Stage-1 map = 24%, reduce = 0%
2013-09-18 00:09:27,313 Stage-1 map = 27%, reduce = 0%
2013-09-18 00:09:28,319 Stage-1 map = 34%, reduce = 0%
2013-09-18 00:09:29,325 Stage-1 map = 37%, reduce = 0%
2013-09-18 00:09:30,337 Stage-1 map = 38%, reduce = 0%, Cumulative CPU 616.5 sec
2013-09-18 00:09:31,344 Stage-1 map = 42%, reduce = 0%, Cumulative CPU 616.5 sec
2013-09-18 00:09:32,351 Stage-1 map = 45%, reduce = 0%, Cumulative CPU 616.5 sec
2013-09-18 00:09:33,366 Stage-1 map = 46%, reduce = 0%, Cumulative CPU 616.5 sec
2013-09-18 00:09:34,373 Stage-1 map = 48%, reduce = 0%, Cumulative CPU 616.5 sec
2013-09-18 00:09:35,382 Stage-1 map = 55%, reduce = 0%, Cumulative CPU 628.21 sec
2013-09-18 00:09:36,418 Stage-1 map = 55%, reduce = 0%, Cumulative CPU 628.21 sec
2013-09-18 00:09:37,437 Stage-1 map = 60%, reduce = 0%, Cumulative CPU 628.21 sec
2013-09-18 00:09:38,446 Stage-1 map = 62%, reduce = 0%, Cumulative CPU 628.21 sec
2013-09-18 00:09:39,465 Stage-1 map = 63%, reduce = 0%, Cumulative CPU 628.21 sec
2013-09-18 00:09:40,473 Stage-1 map = 67%, reduce = 0%, Cumulative CPU 628.21 sec
2013-09-18 00:09:41,481 Stage-1 map = 71%, reduce = 0%, Cumulative CPU 628.21 sec
2013-09-18 00:09:42,488 Stage-1 map = 71%, reduce = 0%, Cumulative CPU 641.75 sec
2013-09-18 00:09:43,500 Stage-1 map = 74%, reduce = 0%, Cumulative CPU 655.91 sec
2013-09-18 00:09:44,507 Stage-1 map = 79%, reduce = 0%, Cumulative CPU 655.91 sec
2013-09-18 00:09:45,524 Stage-1 map = 80%, reduce = 0%, Cumulative CPU 655.91 sec
2013-09-18 00:09:46,548 Stage-1 map = 84%, reduce = 0%, Cumulative CPU 710.09 sec
2013-09-18 00:09:47,560 Stage-1 map = 87%, reduce = 0%, Cumulative CPU 729.24 sec
2013-09-18 00:09:48,567 Stage-1 map = 88%, reduce = 0%, Cumulative CPU 746.21 sec
2013-09-18 00:09:49,578 Stage-1 map = 89%, reduce = 0%, Cumulative CPU 781.66 sec
2013-09-18 00:09:50,585 Stage-1 map = 92%, reduce = 0%, Cumulative CPU 893.41 sec
2013-09-18 00:09:51,590 Stage-1 map = 94%, reduce = 1%, Cumulative CPU 997.39 sec
2013-09-18 00:09:52,799 Stage-1 map = 96%, reduce = 5%, Cumulative CPU 1063.45 sec
2013-09-18 00:09:53,805 Stage-1 map = 98%, reduce = 7%, Cumulative CPU 1196.08 sec
2013-09-18 00:09:54,811 Stage-1 map = 99%, reduce = 12%, Cumulative CPU 1239.77 sec
2013-09-18 00:09:55,817 Stage-1 map = 99%, reduce = 18%, Cumulative CPU 1264.22 sec
2013-09-18 00:09:56,822 Stage-1 map = 99%, reduce = 20%, Cumulative CPU 1289.61 sec
2013-09-18 00:09:57,828 Stage-1 map = 100%, reduce = 21%, Cumulative CPU 1316.56 sec
2013-09-18 00:09:58,847 Stage-1 map = 100%, reduce = 27%, Cumulative CPU 1316.56 sec
2013-09-18 00:09:59,853 Stage-1 map = 100%, reduce = 29%, Cumulative CPU 1316.56 sec
2013-09-18 00:10:00,859 Stage-1 map = 100%, reduce = 30%, Cumulative CPU 1316.56 sec
2013-09-18 00:10:01,866 Stage-1 map = 100%, reduce = 31%, Cumulative CPU 1316.56 sec
2013-09-18 00:10:02,872 Stage-1 map = 100%, reduce = 31%, Cumulative CPU 1316.56 sec
2013-09-18 00:10:03,879 Stage-1 map = 100%, reduce = 32%, Cumulative CPU 1316.56 sec
2013-09-18 00:10:04,885 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 1316.56 sec
2013-09-18 00:10:05,892 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 1316.56 sec
2013-09-18 00:10:06,899 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 1316.56 sec
2013-09-18 00:10:07,908 Stage-1 map = 100%, reduce = 42%, Cumulative CPU 1316.56 sec
2013-09-18 00:10:08,915 Stage-1 map = 100%, reduce = 49%, Cumulative CPU 1316.56 sec
2013-09-18 00:10:09,921 Stage-1 map = 100%, reduce = 58%, Cumulative CPU 1316.56 sec
2013-09-18 00:10:11,093 Stage-1 map = 100%, reduce = 68%, Cumulative CPU 1316.56 sec
2013-09-18 00:10:12,099 Stage-1 map = 100%, reduce = 69%, Cumulative CPU 1316.56 sec
2013-09-18 00:10:13,105 Stage-1 map = 100%, reduce = 69%, Cumulative CPU 1316.56 sec
2013-09-18 00:10:14,111 Stage-1 map = 100%, reduce = 71%, Cumulative CPU 1316.56 sec
2013-09-18 00:10:15,117 Stage-1 map = 100%, reduce = 72%, Cumulative CPU 1316.56 sec
2013-09-18 00:10:19,314 Stage-1 map = 100%, reduce = 74%, Cumulative CPU 1316.56 sec
2013-09-18 00:10:20,320 Stage-1 map = 100%, reduce = 74%, Cumulative CPU 1316.56 sec
2013-09-18 00:10:21,802 Stage-1 map = 100%, reduce = 74%, Cumulative CPU 1316.56 sec
2013-09-18 00:10:22,808 Stage-1 map = 100%, reduce = 74%, Cumulative CPU 1316.56 sec
2013-09-18 00:10:23,814 Stage-1 map = 100%, reduce = 74%, Cumulative CPU 1316.56 sec
2013-09-18 00:10:24,820 Stage-1 map = 100%, reduce = 76%, Cumulative CPU 1316.56 sec
2013-09-18 00:10:25,825 Stage-1 map = 100%, reduce = 76%, Cumulative CPU 1316.56 sec
2013-09-18 00:10:27,283 Stage-1 map = 100%, reduce = 76%, Cumulative CPU 1316.56 sec
2013-09-18 00:10:28,288 Stage-1 map = 100%, reduce = 76%, Cumulative CPU 1316.56 sec
2013-09-18 00:10:29,293 Stage-1 map = 100%, reduce = 76%, Cumulative CPU 1316.56 sec
2013-09-18 00:10:30,301 Stage-1 map = 100%, reduce = 80%, Cumulative CPU 1610.02 sec
2013-09-18 00:10:31,307 Stage-1 map = 100%, reduce = 80%, Cumulative CPU 1610.02 sec
2013-09-18 00:10:32,316 Stage-1 map = 100%, reduce = 80%, Cumulative CPU 1610.02 sec
2013-09-18 00:10:33,323 Stage-1 map = 100%, reduce = 84%, Cumulative CPU 1610.02 sec
2013-09-18 00:10:34,609 Stage-1 map = 100%, reduce = 84%, Cumulative CPU 1610.02 sec
2013-09-18 00:10:35,615 Stage-1 map = 100%, reduce = 84%, Cumulative CPU 1610.02 sec
2013-09-18 00:10:36,620 Stage-1 map = 100%, reduce = 88%, Cumulative CPU 1610.02 sec
2013-09-18 00:10:37,750 Stage-1 map = 100%, reduce = 88%, Cumulative CPU 1610.02 sec
2013-09-18 00:10:38,756 Stage-1 map = 100%, reduce = 91%, Cumulative CPU 1610.02 sec
2013-09-18 00:10:39,762 Stage-1 map = 100%, reduce = 91%, Cumulative CPU 1610.02 sec
2013-09-18 00:10:40,767 Stage-1 map = 100%, reduce = 91%, Cumulative CPU 1610.02 sec
2013-09-18 00:10:41,772 Stage-1 map = 100%, reduce = 94%, Cumulative CPU 1610.02 sec
2013-09-18 00:10:43,972 Stage-1 map = 100%, reduce = 94%, Cumulative CPU 1610.02 sec
2013-09-18 00:10:44,978 Stage-1 map = 100%, reduce = 97%, Cumulative CPU 1610.02 sec
2013-09-18 00:10:45,984 Stage-1 map = 100%, reduce = 97%, Cumulative CPU 1610.02 sec
2013-09-18 00:10:46,990 Stage-1 map = 100%, reduce = 97%, Cumulative CPU 1610.02 sec
2013-09-18 00:10:48,166 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1610.02 sec
2013-09-18 00:10:49,171 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1610.02 sec
2013-09-18 00:10:50,176 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1610.02 sec
2013-09-18 00:10:51,182 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1776.03 sec
2013-09-18 00:10:52,188 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 1795.3 sec
2013-09-18 00:10:53,193 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 1795.3 sec
MapReduce Total cumulative CPU time: 29 minutes 55 seconds 300 msec
Ended Job = job_201309172235_0088
Launching Job 2 out of 2
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0089
Hadoop job information for Stage-2: number of mappers: 9; number of reducers: 1
2013-09-18 00:10:56,790 Stage-2 map = 0%, reduce = 0%
2013-09-18 00:11:13,933 Stage-2 map = 22%, reduce = 0%, Cumulative CPU 22.42 sec
2013-09-18 00:11:14,937 Stage-2 map = 22%, reduce = 0%, Cumulative CPU 22.42 sec
2013-09-18 00:11:15,942 Stage-2 map = 22%, reduce = 0%, Cumulative CPU 22.42 sec
2013-09-18 00:11:16,948 Stage-2 map = 33%, reduce = 0%, Cumulative CPU 22.42 sec
2013-09-18 00:11:17,953 Stage-2 map = 33%, reduce = 0%, Cumulative CPU 22.42 sec
2013-09-18 00:11:18,959 Stage-2 map = 33%, reduce = 0%, Cumulative CPU 22.42 sec
2013-09-18 00:11:19,964 Stage-2 map = 33%, reduce = 0%, Cumulative CPU 22.42 sec
2013-09-18 00:11:20,969 Stage-2 map = 33%, reduce = 0%, Cumulative CPU 22.42 sec
2013-09-18 00:11:21,974 Stage-2 map = 33%, reduce = 4%, Cumulative CPU 22.42 sec
2013-09-18 00:11:22,980 Stage-2 map = 47%, reduce = 4%, Cumulative CPU 22.42 sec
2013-09-18 00:11:23,985 Stage-2 map = 47%, reduce = 4%, Cumulative CPU 22.42 sec
2013-09-18 00:11:24,991 Stage-2 map = 47%, reduce = 4%, Cumulative CPU 22.42 sec
2013-09-18 00:11:25,996 Stage-2 map = 56%, reduce = 4%, Cumulative CPU 22.42 sec
2013-09-18 00:11:27,001 Stage-2 map = 56%, reduce = 4%, Cumulative CPU 22.42 sec
2013-09-18 00:11:28,007 Stage-2 map = 56%, reduce = 4%, Cumulative CPU 22.42 sec
2013-09-18 00:11:29,012 Stage-2 map = 56%, reduce = 4%, Cumulative CPU 22.42 sec
2013-09-18 00:11:30,017 Stage-2 map = 56%, reduce = 4%, Cumulative CPU 339.47 sec
2013-09-18 00:11:31,022 Stage-2 map = 56%, reduce = 4%, Cumulative CPU 339.47 sec
2013-09-18 00:11:32,027 Stage-2 map = 67%, reduce = 4%, Cumulative CPU 339.47 sec
2013-09-18 00:11:33,031 Stage-2 map = 67%, reduce = 4%, Cumulative CPU 339.47 sec
2013-09-18 00:11:34,036 Stage-2 map = 67%, reduce = 4%, Cumulative CPU 339.47 sec
2013-09-18 00:11:35,040 Stage-2 map = 78%, reduce = 4%, Cumulative CPU 339.47 sec
2013-09-18 00:11:36,045 Stage-2 map = 78%, reduce = 4%, Cumulative CPU 339.47 sec
2013-09-18 00:11:37,050 Stage-2 map = 78%, reduce = 4%, Cumulative CPU 339.47 sec
2013-09-18 00:11:38,080 Stage-2 map = 78%, reduce = 4%, Cumulative CPU 339.47 sec
2013-09-18 00:11:39,085 Stage-2 map = 78%, reduce = 4%, Cumulative CPU 339.47 sec
2013-09-18 00:11:40,099 Stage-2 map = 78%, reduce = 4%, Cumulative CPU 339.47 sec
2013-09-18 00:11:41,104 Stage-2 map = 78%, reduce = 4%, Cumulative CPU 339.47 sec
2013-09-18 00:11:42,109 Stage-2 map = 78%, reduce = 4%, Cumulative CPU 339.47 sec
2013-09-18 00:11:43,114 Stage-2 map = 78%, reduce = 4%, Cumulative CPU 339.47 sec
2013-09-18 00:11:44,120 Stage-2 map = 78%, reduce = 4%, Cumulative CPU 339.47 sec
2013-09-18 00:11:45,125 Stage-2 map = 78%, reduce = 4%, Cumulative CPU 339.47 sec
2013-09-18 00:11:46,130 Stage-2 map = 81%, reduce = 4%, Cumulative CPU 358.69 sec
2013-09-18 00:11:47,135 Stage-2 map = 89%, reduce = 4%, Cumulative CPU 418.36 sec
2013-09-18 00:11:48,139 Stage-2 map = 89%, reduce = 4%, Cumulative CPU 418.36 sec
2013-09-18 00:11:49,144 Stage-2 map = 89%, reduce = 4%, Cumulative CPU 418.36 sec
2013-09-18 00:11:50,150 Stage-2 map = 100%, reduce = 4%, Cumulative CPU 510.55 sec
2013-09-18 00:11:51,155 Stage-2 map = 100%, reduce = 4%, Cumulative CPU 510.55 sec
2013-09-18 00:11:52,160 Stage-2 map = 100%, reduce = 4%, Cumulative CPU 510.55 sec
2013-09-18 00:11:53,164 Stage-2 map = 100%, reduce = 4%, Cumulative CPU 510.55 sec
2013-09-18 00:11:54,169 Stage-2 map = 100%, reduce = 4%, Cumulative CPU 510.55 sec
2013-09-18 00:11:55,174 Stage-2 map = 100%, reduce = 19%, Cumulative CPU 510.55 sec
2013-09-18 00:11:56,178 Stage-2 map = 100%, reduce = 19%, Cumulative CPU 510.55 sec
2013-09-18 00:11:57,181 Stage-2 map = 100%, reduce = 19%, Cumulative CPU 510.55 sec
2013-09-18 00:11:58,185 Stage-2 map = 100%, reduce = 19%, Cumulative CPU 510.55 sec
2013-09-18 00:11:59,189 Stage-2 map = 100%, reduce = 19%, Cumulative CPU 510.55 sec
2013-09-18 00:12:00,194 Stage-2 map = 100%, reduce = 19%, Cumulative CPU 510.55 sec
2013-09-18 00:12:01,199 Stage-2 map = 100%, reduce = 22%, Cumulative CPU 510.55 sec
2013-09-18 00:12:02,202 Stage-2 map = 100%, reduce = 22%, Cumulative CPU 510.55 sec
2013-09-18 00:12:03,206 Stage-2 map = 100%, reduce = 22%, Cumulative CPU 510.55 sec
2013-09-18 00:12:04,210 Stage-2 map = 100%, reduce = 26%, Cumulative CPU 510.55 sec
2013-09-18 00:12:05,214 Stage-2 map = 100%, reduce = 26%, Cumulative CPU 510.55 sec
2013-09-18 00:12:06,218 Stage-2 map = 100%, reduce = 26%, Cumulative CPU 510.55 sec
2013-09-18 00:12:07,222 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 510.55 sec
2013-09-18 00:12:08,226 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 510.55 sec
2013-09-18 00:12:09,230 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 510.55 sec
2013-09-18 00:12:10,234 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 510.55 sec
2013-09-18 00:12:11,238 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 510.55 sec
2013-09-18 00:12:12,241 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 510.55 sec
2013-09-18 00:12:13,245 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 510.55 sec
2013-09-18 00:12:14,250 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 510.55 sec
2013-09-18 00:12:15,253 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 510.55 sec
2013-09-18 00:12:16,257 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 510.55 sec
2013-09-18 00:12:17,261 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 510.55 sec
2013-09-18 00:12:18,265 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 510.55 sec
2013-09-18 00:12:19,826 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 510.55 sec
2013-09-18 00:12:20,830 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 510.55 sec
2013-09-18 00:12:21,833 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 510.55 sec
2013-09-18 00:12:24,804 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 510.55 sec
2013-09-18 00:12:31,735 Stage-2 map = 100%, reduce = 67%, Cumulative CPU 545.43 sec
2013-09-18 00:12:32,740 Stage-2 map = 100%, reduce = 67%, Cumulative CPU 545.43 sec
2013-09-18 00:12:33,744 Stage-2 map = 100%, reduce = 67%, Cumulative CPU 545.43 sec
2013-09-18 00:12:34,748 Stage-2 map = 100%, reduce = 71%, Cumulative CPU 545.43 sec
2013-09-18 00:12:35,753 Stage-2 map = 100%, reduce = 71%, Cumulative CPU 545.43 sec
2013-09-18 00:12:36,757 Stage-2 map = 100%, reduce = 71%, Cumulative CPU 545.43 sec
2013-09-18 00:12:37,762 Stage-2 map = 100%, reduce = 76%, Cumulative CPU 545.43 sec
2013-09-18 00:12:39,183 Stage-2 map = 100%, reduce = 76%, Cumulative CPU 545.43 sec
2013-09-18 00:12:40,188 Stage-2 map = 100%, reduce = 76%, Cumulative CPU 545.43 sec
2013-09-18 00:12:41,192 Stage-2 map = 100%, reduce = 76%, Cumulative CPU 545.43 sec
2013-09-18 00:12:42,196 Stage-2 map = 100%, reduce = 76%, Cumulative CPU 545.43 sec
2013-09-18 00:12:43,622 Stage-2 map = 100%, reduce = 76%, Cumulative CPU 545.43 sec
2013-09-18 00:12:44,627 Stage-2 map = 100%, reduce = 76%, Cumulative CPU 545.43 sec
2013-09-18 00:12:45,631 Stage-2 map = 100%, reduce = 76%, Cumulative CPU 545.43 sec
2013-09-18 00:12:46,636 Stage-2 map = 100%, reduce = 76%, Cumulative CPU 545.43 sec
2013-09-18 00:12:47,640 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 559.08 sec
2013-09-18 00:12:48,644 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 559.08 sec
2013-09-18 00:12:49,648 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 559.08 sec
MapReduce Total cumulative CPU time: 9 minutes 19 seconds 80 msec
Ended Job = job_201309172235_0089
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 11 Cumulative CPU: 1795.3 sec HDFS Read: 843455069 HDFS Write: 2238428518 SUCCESS
Job 1: Map: 9 Reduce: 1 Cumulative CPU: 559.08 sec HDFS Read: 2238513761 HDFS Write: 278 SUCCESS
Total MapReduce CPU Time Spent: 39 minutes 14 seconds 380 msec
OK
Time taken: 237.554 seconds, Fetched: 10 row(s)
hive> quit;
times: 2
query: SELECT UserID, minute(EventTime), SearchPhrase, count(*) AS c FROM hits_100m GROUP BY UserID, minute(EventTime), SearchPhrase ORDER BY c DESC LIMIT 10;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_21185@mturlrep13_201309180012_506617682.txt
hive> SELECT UserID, minute(EventTime), SearchPhrase, count(*) AS c FROM hits_100m GROUP BY UserID, minute(EventTime), SearchPhrase ORDER BY c DESC LIMIT 10;;
Total MapReduce jobs = 2
Launching Job 1 out of 2
Number of reduce tasks not specified. Estimated from input data size: 11
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0090
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 11
2013-09-18 00:12:59,255 Stage-1 map = 0%, reduce = 0%
2013-09-18 00:13:15,399 Stage-1 map = 2%, reduce = 0%
2013-09-18 00:13:16,405 Stage-1 map = 8%, reduce = 0%
2013-09-18 00:13:18,423 Stage-1 map = 12%, reduce = 0%
2013-09-18 00:13:19,429 Stage-1 map = 19%, reduce = 0%
2013-09-18 00:13:21,449 Stage-1 map = 21%, reduce = 0%
2013-09-18 00:13:22,454 Stage-1 map = 31%, reduce = 0%
2013-09-18 00:13:23,475 Stage-1 map = 32%, reduce = 0%
2013-09-18 00:13:25,512 Stage-1 map = 40%, reduce = 0%
2013-09-18 00:13:26,534 Stage-1 map = 41%, reduce = 0%
2013-09-18 00:13:28,545 Stage-1 map = 50%, reduce = 0%
2013-09-18 00:13:29,557 Stage-1 map = 52%, reduce = 0%
2013-09-18 00:13:30,834 Stage-1 map = 53%, reduce = 0%, Cumulative CPU 21.31 sec
2013-09-18 00:13:31,841 Stage-1 map = 61%, reduce = 0%, Cumulative CPU 814.57 sec
2013-09-18 00:13:32,847 Stage-1 map = 62%, reduce = 0%, Cumulative CPU 814.57 sec
2013-09-18 00:13:33,857 Stage-1 map = 63%, reduce = 0%, Cumulative CPU 816.31 sec
2013-09-18 00:13:34,872 Stage-1 map = 68%, reduce = 0%, Cumulative CPU 816.31 sec
2013-09-18 00:13:35,886 Stage-1 map = 71%, reduce = 0%, Cumulative CPU 816.31 sec
2013-09-18 00:13:36,894 Stage-1 map = 71%, reduce = 0%, Cumulative CPU 816.31 sec
2013-09-18 00:13:37,900 Stage-1 map = 77%, reduce = 0%, Cumulative CPU 824.05 sec
2013-09-18 00:13:38,913 Stage-1 map = 79%, reduce = 0%, Cumulative CPU 830.96 sec
2013-09-18 00:13:39,925 Stage-1 map = 79%, reduce = 0%, Cumulative CPU 830.96 sec
2013-09-18 00:13:40,931 Stage-1 map = 83%, reduce = 0%, Cumulative CPU 830.96 sec
2013-09-18 00:13:41,938 Stage-1 map = 85%, reduce = 0%, Cumulative CPU 830.96 sec
2013-09-18 00:13:42,978 Stage-1 map = 87%, reduce = 0%, Cumulative CPU 865.43 sec
2013-09-18 00:13:43,986 Stage-1 map = 88%, reduce = 0%, Cumulative CPU 875.5 sec
2013-09-18 00:13:44,998 Stage-1 map = 90%, reduce = 0%, Cumulative CPU 908.75 sec
2013-09-18 00:13:46,004 Stage-1 map = 93%, reduce = 0%, Cumulative CPU 1001.65 sec
2013-09-18 00:13:47,011 Stage-1 map = 94%, reduce = 0%, Cumulative CPU 1063.43 sec
2013-09-18 00:13:48,018 Stage-1 map = 98%, reduce = 0%, Cumulative CPU 1202.46 sec
2013-09-18 00:13:49,023 Stage-1 map = 99%, reduce = 0%, Cumulative CPU 1255.44 sec
2013-09-18 00:13:50,029 Stage-1 map = 99%, reduce = 3%, Cumulative CPU 1272.82 sec
2013-09-18 00:13:51,035 Stage-1 map = 99%, reduce = 7%, Cumulative CPU 1272.82 sec
2013-09-18 00:13:52,040 Stage-1 map = 100%, reduce = 18%, Cumulative CPU 1316.55 sec
2013-09-18 00:13:53,046 Stage-1 map = 100%, reduce = 23%, Cumulative CPU 1316.55 sec
2013-09-18 00:13:54,052 Stage-1 map = 100%, reduce = 27%, Cumulative CPU 1316.55 sec
2013-09-18 00:13:55,065 Stage-1 map = 100%, reduce = 31%, Cumulative CPU 1316.55 sec
2013-09-18 00:13:56,071 Stage-1 map = 100%, reduce = 32%, Cumulative CPU 1316.55 sec
2013-09-18 00:13:57,077 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 1316.55 sec
2013-09-18 00:13:58,083 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 1316.55 sec
2013-09-18 00:13:59,109 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 1316.55 sec
2013-09-18 00:14:00,116 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 1316.55 sec
2013-09-18 00:14:01,122 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 1316.55 sec
2013-09-18 00:14:02,127 Stage-1 map = 100%, reduce = 36%, Cumulative CPU 1316.55 sec
2013-09-18 00:14:03,133 Stage-1 map = 100%, reduce = 40%, Cumulative CPU 1316.55 sec
2013-09-18 00:14:04,139 Stage-1 map = 100%, reduce = 40%, Cumulative CPU 1316.55 sec
2013-09-18 00:14:05,145 Stage-1 map = 100%, reduce = 43%, Cumulative CPU 1316.55 sec
2013-09-18 00:14:06,152 Stage-1 map = 100%, reduce = 50%, Cumulative CPU 1316.55 sec
2013-09-18 00:14:07,158 Stage-1 map = 100%, reduce = 59%, Cumulative CPU 1316.55 sec
2013-09-18 00:14:08,164 Stage-1 map = 100%, reduce = 66%, Cumulative CPU 1316.55 sec
2013-09-18 00:14:09,702 Stage-1 map = 100%, reduce = 70%, Cumulative CPU 1316.55 sec
2013-09-18 00:14:10,707 Stage-1 map = 100%, reduce = 70%, Cumulative CPU 1316.55 sec
2013-09-18 00:14:11,713 Stage-1 map = 100%, reduce = 70%, Cumulative CPU 1316.55 sec
2013-09-18 00:14:12,719 Stage-1 map = 100%, reduce = 71%, Cumulative CPU 1316.55 sec
2013-09-18 00:14:13,725 Stage-1 map = 100%, reduce = 72%, Cumulative CPU 1316.55 sec
2013-09-18 00:14:14,731 Stage-1 map = 100%, reduce = 72%, Cumulative CPU 1316.55 sec
2013-09-18 00:14:15,959 Stage-1 map = 100%, reduce = 74%, Cumulative CPU 1316.55 sec
2013-09-18 00:14:16,964 Stage-1 map = 100%, reduce = 75%, Cumulative CPU 1316.55 sec
2013-09-18 00:14:17,970 Stage-1 map = 100%, reduce = 75%, Cumulative CPU 1316.55 sec
2013-09-18 00:14:18,976 Stage-1 map = 100%, reduce = 75%, Cumulative CPU 1316.55 sec
2013-09-18 00:14:19,982 Stage-1 map = 100%, reduce = 77%, Cumulative CPU 1316.55 sec
2013-09-18 00:14:21,672 Stage-1 map = 100%, reduce = 78%, Cumulative CPU 1316.55 sec
2013-09-18 00:14:24,777 Stage-1 map = 100%, reduce = 79%, Cumulative CPU 1316.55 sec
2013-09-18 00:14:25,782 Stage-1 map = 100%, reduce = 79%, Cumulative CPU 1316.55 sec
2013-09-18 00:14:26,787 Stage-1 map = 100%, reduce = 80%, Cumulative CPU 1316.55 sec
2013-09-18 00:14:30,291 Stage-1 map = 100%, reduce = 81%, Cumulative CPU 1316.55 sec
2013-09-18 00:14:34,570 Stage-1 map = 100%, reduce = 83%, Cumulative CPU 1638.29 sec
2013-09-18 00:14:35,576 Stage-1 map = 100%, reduce = 83%, Cumulative CPU 1638.29 sec
2013-09-18 00:14:36,582 Stage-1 map = 100%, reduce = 84%, Cumulative CPU 1638.29 sec
2013-09-18 00:14:37,588 Stage-1 map = 100%, reduce = 88%, Cumulative CPU 1638.29 sec
2013-09-18 00:14:38,594 Stage-1 map = 100%, reduce = 88%, Cumulative CPU 1638.29 sec
2013-09-18 00:14:39,628 Stage-1 map = 100%, reduce = 88%, Cumulative CPU 1638.29 sec
2013-09-18 00:14:40,634 Stage-1 map = 100%, reduce = 88%, Cumulative CPU 1638.29 sec
2013-09-18 00:14:41,640 Stage-1 map = 100%, reduce = 88%, Cumulative CPU 1638.29 sec
2013-09-18 00:14:42,645 Stage-1 map = 100%, reduce = 92%, Cumulative CPU 1638.29 sec
2013-09-18 00:14:43,651 Stage-1 map = 100%, reduce = 92%, Cumulative CPU 1638.29 sec
2013-09-18 00:14:44,657 Stage-1 map = 100%, reduce = 92%, Cumulative CPU 1638.29 sec
2013-09-18 00:14:47,811 Stage-1 map = 100%, reduce = 96%, Cumulative CPU 1648.84 sec
2013-09-18 00:14:48,817 Stage-1 map = 100%, reduce = 98%, Cumulative CPU 1725.98 sec
2013-09-18 00:14:49,823 Stage-1 map = 100%, reduce = 98%, Cumulative CPU 1725.98 sec
2013-09-18 00:14:54,762 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1725.98 sec
2013-09-18 00:14:56,039 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 1784.82 sec
2013-09-18 00:14:57,044 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 1784.82 sec
MapReduce Total cumulative CPU time: 29 minutes 44 seconds 820 msec
Ended Job = job_201309172235_0090
Launching Job 2 out of 2
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0091
Hadoop job information for Stage-2: number of mappers: 9; number of reducers: 1
2013-09-18 00:15:00,513 Stage-2 map = 0%, reduce = 0%
2013-09-18 00:15:17,637 Stage-2 map = 30%, reduce = 0%
2013-09-18 00:15:18,642 Stage-2 map = 31%, reduce = 0%, Cumulative CPU 21.69 sec
2013-09-18 00:15:19,647 Stage-2 map = 31%, reduce = 0%, Cumulative CPU 21.69 sec
2013-09-18 00:15:20,652 Stage-2 map = 33%, reduce = 0%, Cumulative CPU 21.69 sec
2013-09-18 00:15:21,657 Stage-2 map = 33%, reduce = 0%, Cumulative CPU 21.69 sec
2013-09-18 00:15:22,662 Stage-2 map = 33%, reduce = 0%, Cumulative CPU 21.69 sec
2013-09-18 00:15:23,667 Stage-2 map = 33%, reduce = 0%, Cumulative CPU 21.69 sec
2013-09-18 00:15:24,672 Stage-2 map = 33%, reduce = 0%, Cumulative CPU 21.69 sec
2013-09-18 00:15:25,678 Stage-2 map = 33%, reduce = 4%, Cumulative CPU 21.69 sec
2013-09-18 00:15:26,683 Stage-2 map = 47%, reduce = 4%, Cumulative CPU 21.69 sec
2013-09-18 00:15:27,687 Stage-2 map = 47%, reduce = 4%, Cumulative CPU 21.69 sec
2013-09-18 00:15:28,692 Stage-2 map = 47%, reduce = 4%, Cumulative CPU 21.69 sec
2013-09-18 00:15:29,697 Stage-2 map = 56%, reduce = 4%, Cumulative CPU 21.69 sec
2013-09-18 00:15:30,702 Stage-2 map = 56%, reduce = 4%, Cumulative CPU 21.69 sec
2013-09-18 00:15:31,707 Stage-2 map = 56%, reduce = 4%, Cumulative CPU 21.69 sec
2013-09-18 00:15:32,712 Stage-2 map = 56%, reduce = 4%, Cumulative CPU 21.69 sec
2013-09-18 00:15:33,717 Stage-2 map = 56%, reduce = 4%, Cumulative CPU 21.69 sec
2013-09-18 00:15:34,722 Stage-2 map = 56%, reduce = 4%, Cumulative CPU 343.13 sec
2013-09-18 00:15:35,728 Stage-2 map = 70%, reduce = 4%, Cumulative CPU 343.13 sec
2013-09-18 00:15:36,733 Stage-2 map = 70%, reduce = 4%, Cumulative CPU 343.13 sec
2013-09-18 00:15:37,738 Stage-2 map = 70%, reduce = 4%, Cumulative CPU 343.13 sec
2013-09-18 00:15:38,744 Stage-2 map = 72%, reduce = 4%, Cumulative CPU 343.13 sec
2013-09-18 00:15:39,749 Stage-2 map = 78%, reduce = 4%, Cumulative CPU 343.13 sec
2013-09-18 00:15:40,754 Stage-2 map = 78%, reduce = 4%, Cumulative CPU 343.13 sec
2013-09-18 00:15:41,906 Stage-2 map = 78%, reduce = 4%, Cumulative CPU 343.13 sec
2013-09-18 00:15:42,911 Stage-2 map = 78%, reduce = 4%, Cumulative CPU 343.13 sec
2013-09-18 00:15:43,916 Stage-2 map = 78%, reduce = 4%, Cumulative CPU 343.13 sec
2013-09-18 00:15:44,921 Stage-2 map = 78%, reduce = 4%, Cumulative CPU 343.13 sec
2013-09-18 00:15:45,951 Stage-2 map = 78%, reduce = 4%, Cumulative CPU 343.13 sec
2013-09-18 00:15:46,956 Stage-2 map = 78%, reduce = 4%, Cumulative CPU 343.13 sec
2013-09-18 00:15:47,961 Stage-2 map = 78%, reduce = 4%, Cumulative CPU 343.13 sec
2013-09-18 00:15:48,977 Stage-2 map = 78%, reduce = 4%, Cumulative CPU 343.13 sec
2013-09-18 00:15:49,982 Stage-2 map = 78%, reduce = 4%, Cumulative CPU 343.13 sec
2013-09-18 00:15:50,988 Stage-2 map = 78%, reduce = 4%, Cumulative CPU 343.13 sec
2013-09-18 00:15:52,053 Stage-2 map = 78%, reduce = 4%, Cumulative CPU 343.13 sec
2013-09-18 00:15:53,058 Stage-2 map = 92%, reduce = 4%, Cumulative CPU 444.1 sec
2013-09-18 00:15:54,063 Stage-2 map = 94%, reduce = 4%, Cumulative CPU 466.73 sec
2013-09-18 00:15:55,109 Stage-2 map = 100%, reduce = 4%, Cumulative CPU 513.25 sec
2013-09-18 00:15:56,114 Stage-2 map = 100%, reduce = 4%, Cumulative CPU 513.25 sec
2013-09-18 00:15:57,119 Stage-2 map = 100%, reduce = 4%, Cumulative CPU 513.25 sec
2013-09-18 00:15:58,124 Stage-2 map = 100%, reduce = 4%, Cumulative CPU 513.25 sec
2013-09-18 00:15:59,129 Stage-2 map = 100%, reduce = 4%, Cumulative CPU 513.25 sec
2013-09-18 00:16:00,133 Stage-2 map = 100%, reduce = 4%, Cumulative CPU 513.25 sec
2013-09-18 00:16:01,138 Stage-2 map = 100%, reduce = 19%, Cumulative CPU 513.25 sec
2013-09-18 00:16:02,142 Stage-2 map = 100%, reduce = 19%, Cumulative CPU 513.25 sec
2013-09-18 00:16:03,148 Stage-2 map = 100%, reduce = 19%, Cumulative CPU 513.25 sec
2013-09-18 00:16:04,152 Stage-2 map = 100%, reduce = 19%, Cumulative CPU 513.25 sec
2013-09-18 00:16:05,157 Stage-2 map = 100%, reduce = 19%, Cumulative CPU 513.25 sec
2013-09-18 00:16:06,161 Stage-2 map = 100%, reduce = 19%, Cumulative CPU 513.25 sec
2013-09-18 00:16:07,165 Stage-2 map = 100%, reduce = 26%, Cumulative CPU 513.25 sec
2013-09-18 00:16:08,170 Stage-2 map = 100%, reduce = 26%, Cumulative CPU 513.25 sec
2013-09-18 00:16:09,174 Stage-2 map = 100%, reduce = 26%, Cumulative CPU 513.25 sec
2013-09-18 00:16:10,179 Stage-2 map = 100%, reduce = 30%, Cumulative CPU 513.25 sec
2013-09-18 00:16:11,183 Stage-2 map = 100%, reduce = 30%, Cumulative CPU 513.25 sec
2013-09-18 00:16:12,188 Stage-2 map = 100%, reduce = 30%, Cumulative CPU 513.25 sec
2013-09-18 00:16:13,191 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 513.25 sec
2013-09-18 00:16:14,196 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 513.25 sec
2013-09-18 00:16:15,200 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 513.25 sec
2013-09-18 00:16:16,204 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 513.25 sec
2013-09-18 00:16:18,269 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 513.25 sec
2013-09-18 00:16:19,273 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 513.25 sec
2013-09-18 00:16:20,276 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 513.25 sec
2013-09-18 00:16:21,281 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 513.25 sec
2013-09-18 00:16:27,554 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 513.25 sec
2013-09-18 00:16:28,559 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 513.25 sec
2013-09-18 00:16:29,564 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 513.25 sec
2013-09-18 00:16:33,769 Stage-2 map = 100%, reduce = 67%, Cumulative CPU 513.25 sec
2013-09-18 00:16:34,774 Stage-2 map = 100%, reduce = 71%, Cumulative CPU 552.24 sec
2013-09-18 00:16:35,778 Stage-2 map = 100%, reduce = 71%, Cumulative CPU 552.24 sec
2013-09-18 00:16:36,782 Stage-2 map = 100%, reduce = 76%, Cumulative CPU 552.24 sec
2013-09-18 00:16:37,787 Stage-2 map = 100%, reduce = 76%, Cumulative CPU 552.24 sec
2013-09-18 00:16:38,792 Stage-2 map = 100%, reduce = 76%, Cumulative CPU 552.24 sec
2013-09-18 00:16:39,796 Stage-2 map = 100%, reduce = 76%, Cumulative CPU 552.24 sec
2013-09-18 00:16:40,801 Stage-2 map = 100%, reduce = 76%, Cumulative CPU 552.24 sec
2013-09-18 00:16:41,805 Stage-2 map = 100%, reduce = 76%, Cumulative CPU 552.24 sec
2013-09-18 00:16:42,809 Stage-2 map = 100%, reduce = 76%, Cumulative CPU 552.24 sec
2013-09-18 00:16:43,814 Stage-2 map = 100%, reduce = 76%, Cumulative CPU 552.24 sec
2013-09-18 00:16:44,819 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 562.94 sec
2013-09-18 00:16:45,823 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 562.94 sec
MapReduce Total cumulative CPU time: 9 minutes 22 seconds 940 msec
Ended Job = job_201309172235_0091
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 11 Cumulative CPU: 1784.82 sec HDFS Read: 843455069 HDFS Write: 2238428518 SUCCESS
Job 1: Map: 9 Reduce: 1 Cumulative CPU: 562.94 sec HDFS Read: 2238513761 HDFS Write: 278 SUCCESS
Total MapReduce CPU Time Spent: 39 minutes 7 seconds 760 msec
OK
Time taken: 234.361 seconds, Fetched: 10 row(s)
hive> quit;
times: 3
query: SELECT UserID, minute(EventTime), SearchPhrase, count(*) AS c FROM hits_100m GROUP BY UserID, minute(EventTime), SearchPhrase ORDER BY c DESC LIMIT 10;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_30407@mturlrep13_201309180016_1665474298.txt
hive> SELECT UserID, minute(EventTime), SearchPhrase, count(*) AS c FROM hits_100m GROUP BY UserID, minute(EventTime), SearchPhrase ORDER BY c DESC LIMIT 10;;
Total MapReduce jobs = 2
Launching Job 1 out of 2
Number of reduce tasks not specified. Estimated from input data size: 11
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0092
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 11
2013-09-18 00:16:56,793 Stage-1 map = 0%, reduce = 0%
2013-09-18 00:17:10,896 Stage-1 map = 1%, reduce = 0%
2013-09-18 00:17:11,902 Stage-1 map = 4%, reduce = 0%
2013-09-18 00:17:12,907 Stage-1 map = 5%, reduce = 0%
2013-09-18 00:17:13,917 Stage-1 map = 10%, reduce = 0%
2013-09-18 00:17:14,928 Stage-1 map = 15%, reduce = 0%
2013-09-18 00:17:15,933 Stage-1 map = 17%, reduce = 0%
2013-09-18 00:17:16,939 Stage-1 map = 20%, reduce = 0%
2013-09-18 00:17:17,944 Stage-1 map = 24%, reduce = 0%
2013-09-18 00:17:18,950 Stage-1 map = 27%, reduce = 0%
2013-09-18 00:17:19,955 Stage-1 map = 31%, reduce = 0%
2013-09-18 00:17:20,961 Stage-1 map = 33%, reduce = 0%
2013-09-18 00:17:21,966 Stage-1 map = 37%, reduce = 0%
2013-09-18 00:17:22,971 Stage-1 map = 40%, reduce = 0%
2013-09-18 00:17:24,001 Stage-1 map = 44%, reduce = 0%
2013-09-18 00:17:25,019 Stage-1 map = 47%, reduce = 0%
2013-09-18 00:17:26,029 Stage-1 map = 50%, reduce = 0%
2013-09-18 00:17:27,037 Stage-1 map = 53%, reduce = 0%
2013-09-18 00:17:28,044 Stage-1 map = 58%, reduce = 0%
2013-09-18 00:17:29,062 Stage-1 map = 59%, reduce = 0%, Cumulative CPU 21.75 sec
2013-09-18 00:17:30,082 Stage-1 map = 61%, reduce = 0%, Cumulative CPU 21.75 sec
2013-09-18 00:17:31,094 Stage-1 map = 67%, reduce = 0%, Cumulative CPU 21.75 sec
2013-09-18 00:17:32,114 Stage-1 map = 69%, reduce = 0%, Cumulative CPU 21.75 sec
2013-09-18 00:17:33,129 Stage-1 map = 71%, reduce = 0%, Cumulative CPU 21.75 sec
2013-09-18 00:17:34,140 Stage-1 map = 77%, reduce = 0%, Cumulative CPU 105.0 sec
2013-09-18 00:17:35,156 Stage-1 map = 80%, reduce = 0%, Cumulative CPU 1037.71 sec
2013-09-18 00:17:36,167 Stage-1 map = 81%, reduce = 0%, Cumulative CPU 1037.71 sec
2013-09-18 00:17:37,173 Stage-1 map = 84%, reduce = 0%, Cumulative CPU 1037.71 sec
2013-09-18 00:17:38,180 Stage-1 map = 85%, reduce = 0%, Cumulative CPU 1037.71 sec
2013-09-18 00:17:39,186 Stage-1 map = 85%, reduce = 0%, Cumulative CPU 1037.71 sec
2013-09-18 00:17:40,198 Stage-1 map = 87%, reduce = 0%, Cumulative CPU 1047.34 sec
2013-09-18 00:17:41,214 Stage-1 map = 89%, reduce = 0%, Cumulative CPU 1064.75 sec
2013-09-18 00:17:42,227 Stage-1 map = 91%, reduce = 0%, Cumulative CPU 1113.1 sec
2013-09-18 00:17:43,233 Stage-1 map = 95%, reduce = 0%, Cumulative CPU 1183.03 sec
2013-09-18 00:17:44,240 Stage-1 map = 97%, reduce = 0%, Cumulative CPU 1236.75 sec
2013-09-18 00:17:45,256 Stage-1 map = 99%, reduce = 0%, Cumulative CPU 1266.7 sec
2013-09-18 00:17:46,269 Stage-1 map = 99%, reduce = 0%, Cumulative CPU 1290.62 sec
2013-09-18 00:17:47,277 Stage-1 map = 99%, reduce = 1%, Cumulative CPU 1290.62 sec
2013-09-18 00:17:48,283 Stage-1 map = 99%, reduce = 3%, Cumulative CPU 1304.6 sec
2013-09-18 00:17:49,289 Stage-1 map = 100%, reduce = 11%, Cumulative CPU 1320.25 sec
2013-09-18 00:17:50,296 Stage-1 map = 100%, reduce = 18%, Cumulative CPU 1320.25 sec
2013-09-18 00:17:51,313 Stage-1 map = 100%, reduce = 24%, Cumulative CPU 1320.25 sec
2013-09-18 00:17:52,319 Stage-1 map = 100%, reduce = 30%, Cumulative CPU 1320.25 sec
2013-09-18 00:17:53,325 Stage-1 map = 100%, reduce = 31%, Cumulative CPU 1320.25 sec
2013-09-18 00:17:54,331 Stage-1 map = 100%, reduce = 31%, Cumulative CPU 1320.25 sec
2013-09-18 00:17:55,338 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 1320.25 sec
2013-09-18 00:17:56,344 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 1320.25 sec
2013-09-18 00:17:57,350 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 1320.25 sec
2013-09-18 00:17:58,356 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 1320.25 sec
2013-09-18 00:17:59,362 Stage-1 map = 100%, reduce = 39%, Cumulative CPU 1320.25 sec
2013-09-18 00:18:00,368 Stage-1 map = 100%, reduce = 49%, Cumulative CPU 1320.25 sec
2013-09-18 00:18:01,374 Stage-1 map = 100%, reduce = 58%, Cumulative CPU 1320.25 sec
2013-09-18 00:18:02,381 Stage-1 map = 100%, reduce = 68%, Cumulative CPU 1320.25 sec
2013-09-18 00:18:03,387 Stage-1 map = 100%, reduce = 69%, Cumulative CPU 1320.25 sec
2013-09-18 00:18:04,392 Stage-1 map = 100%, reduce = 69%, Cumulative CPU 1320.25 sec
2013-09-18 00:18:07,247 Stage-1 map = 100%, reduce = 71%, Cumulative CPU 1320.25 sec
2013-09-18 00:18:08,252 Stage-1 map = 100%, reduce = 71%, Cumulative CPU 1320.25 sec
2013-09-18 00:18:09,259 Stage-1 map = 100%, reduce = 71%, Cumulative CPU 1320.25 sec
2013-09-18 00:18:10,266 Stage-1 map = 100%, reduce = 73%, Cumulative CPU 1320.25 sec
2013-09-18 00:18:11,272 Stage-1 map = 100%, reduce = 73%, Cumulative CPU 1320.25 sec
2013-09-18 00:18:12,762 Stage-1 map = 100%, reduce = 73%, Cumulative CPU 1320.25 sec
2013-09-18 00:18:13,767 Stage-1 map = 100%, reduce = 73%, Cumulative CPU 1320.25 sec
2013-09-18 00:18:14,772 Stage-1 map = 100%, reduce = 76%, Cumulative CPU 1320.25 sec
2013-09-18 00:18:15,778 Stage-1 map = 100%, reduce = 76%, Cumulative CPU 1320.25 sec
2013-09-18 00:18:18,513 Stage-1 map = 100%, reduce = 76%, Cumulative CPU 1320.25 sec
2013-09-18 00:18:20,464 Stage-1 map = 100%, reduce = 80%, Cumulative CPU 1320.25 sec
2013-09-18 00:18:21,470 Stage-1 map = 100%, reduce = 80%, Cumulative CPU 1320.25 sec
2013-09-18 00:18:22,477 Stage-1 map = 100%, reduce = 80%, Cumulative CPU 1320.25 sec
2013-09-18 00:18:23,674 Stage-1 map = 100%, reduce = 83%, Cumulative CPU 1320.25 sec
2013-09-18 00:18:24,680 Stage-1 map = 100%, reduce = 83%, Cumulative CPU 1320.25 sec
2013-09-18 00:18:26,496 Stage-1 map = 100%, reduce = 85%, Cumulative CPU 1320.25 sec
2013-09-18 00:18:27,502 Stage-1 map = 100%, reduce = 85%, Cumulative CPU 1320.25 sec
2013-09-18 00:18:28,508 Stage-1 map = 100%, reduce = 85%, Cumulative CPU 1320.25 sec
2013-09-18 00:18:31,532 Stage-1 map = 100%, reduce = 89%, Cumulative CPU 1320.25 sec
2013-09-18 00:18:32,538 Stage-1 map = 100%, reduce = 89%, Cumulative CPU 1320.25 sec
2013-09-18 00:18:33,543 Stage-1 map = 100%, reduce = 89%, Cumulative CPU 1320.25 sec
2013-09-18 00:18:34,555 Stage-1 map = 100%, reduce = 93%, Cumulative CPU 1721.3 sec
2013-09-18 00:18:36,736 Stage-1 map = 100%, reduce = 93%, Cumulative CPU 1721.3 sec
2013-09-18 00:18:37,742 Stage-1 map = 100%, reduce = 93%, Cumulative CPU 1721.3 sec
2013-09-18 00:18:38,747 Stage-1 map = 100%, reduce = 93%, Cumulative CPU 1721.3 sec
2013-09-18 00:18:39,753 Stage-1 map = 100%, reduce = 95%, Cumulative CPU 1721.3 sec
2013-09-18 00:18:40,759 Stage-1 map = 100%, reduce = 95%, Cumulative CPU 1724.11 sec
2013-09-18 00:18:41,765 Stage-1 map = 100%, reduce = 95%, Cumulative CPU 1724.11 sec
2013-09-18 00:18:42,770 Stage-1 map = 100%, reduce = 95%, Cumulative CPU 1724.11 sec
2013-09-18 00:18:43,776 Stage-1 map = 100%, reduce = 95%, Cumulative CPU 1724.11 sec
2013-09-18 00:18:44,782 Stage-1 map = 100%, reduce = 98%, Cumulative CPU 1743.27 sec
2013-09-18 00:18:45,788 Stage-1 map = 100%, reduce = 98%, Cumulative CPU 1754.65 sec
2013-09-18 00:18:48,727 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1754.65 sec
2013-09-18 00:18:49,733 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 1786.03 sec
2013-09-18 00:18:50,739 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 1786.03 sec
2013-09-18 00:18:51,754 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 1786.03 sec
MapReduce Total cumulative CPU time: 29 minutes 46 seconds 30 msec
Ended Job = job_201309172235_0092
Launching Job 2 out of 2
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0093
Hadoop job information for Stage-2: number of mappers: 9; number of reducers: 1
2013-09-18 00:18:55,132 Stage-2 map = 0%, reduce = 0%
2013-09-18 00:19:13,198 Stage-2 map = 33%, reduce = 0%
2013-09-18 00:19:21,228 Stage-2 map = 33%, reduce = 4%, Cumulative CPU 22.55 sec
2013-09-18 00:19:22,233 Stage-2 map = 56%, reduce = 4%, Cumulative CPU 22.55 sec
2013-09-18 00:19:23,238 Stage-2 map = 56%, reduce = 4%, Cumulative CPU 22.55 sec
2013-09-18 00:19:24,243 Stage-2 map = 56%, reduce = 4%, Cumulative CPU 22.55 sec
2013-09-18 00:19:25,248 Stage-2 map = 56%, reduce = 4%, Cumulative CPU 22.55 sec
2013-09-18 00:19:26,252 Stage-2 map = 56%, reduce = 4%, Cumulative CPU 22.55 sec
2013-09-18 00:19:27,257 Stage-2 map = 56%, reduce = 4%, Cumulative CPU 22.55 sec
2013-09-18 00:19:28,262 Stage-2 map = 56%, reduce = 4%, Cumulative CPU 22.55 sec
2013-09-18 00:19:29,267 Stage-2 map = 56%, reduce = 4%, Cumulative CPU 22.55 sec
2013-09-18 00:19:30,271 Stage-2 map = 56%, reduce = 4%, Cumulative CPU 22.55 sec
2013-09-18 00:19:31,276 Stage-2 map = 56%, reduce = 4%, Cumulative CPU 22.55 sec
2013-09-18 00:19:32,280 Stage-2 map = 78%, reduce = 4%, Cumulative CPU 22.55 sec
2013-09-18 00:19:33,285 Stage-2 map = 78%, reduce = 4%, Cumulative CPU 22.55 sec
2013-09-18 00:19:34,290 Stage-2 map = 78%, reduce = 4%, Cumulative CPU 22.55 sec
2013-09-18 00:19:35,295 Stage-2 map = 78%, reduce = 4%, Cumulative CPU 398.54 sec
2013-09-18 00:19:36,299 Stage-2 map = 78%, reduce = 4%, Cumulative CPU 398.54 sec
2013-09-18 00:19:37,305 Stage-2 map = 78%, reduce = 4%, Cumulative CPU 398.54 sec
2013-09-18 00:19:38,310 Stage-2 map = 78%, reduce = 4%, Cumulative CPU 398.54 sec
2013-09-18 00:19:39,315 Stage-2 map = 78%, reduce = 4%, Cumulative CPU 398.54 sec
2013-09-18 00:19:40,320 Stage-2 map = 78%, reduce = 4%, Cumulative CPU 398.54 sec
2013-09-18 00:19:41,326 Stage-2 map = 78%, reduce = 4%, Cumulative CPU 398.54 sec
2013-09-18 00:19:42,331 Stage-2 map = 78%, reduce = 4%, Cumulative CPU 398.54 sec
2013-09-18 00:19:43,349 Stage-2 map = 78%, reduce = 4%, Cumulative CPU 398.54 sec
2013-09-18 00:19:44,354 Stage-2 map = 78%, reduce = 4%, Cumulative CPU 398.54 sec
2013-09-18 00:19:46,244 Stage-2 map = 84%, reduce = 4%, Cumulative CPU 423.18 sec
2013-09-18 00:19:47,248 Stage-2 map = 94%, reduce = 4%, Cumulative CPU 476.82 sec
2013-09-18 00:19:48,253 Stage-2 map = 100%, reduce = 4%, Cumulative CPU 505.58 sec
2013-09-18 00:19:49,258 Stage-2 map = 100%, reduce = 4%, Cumulative CPU 505.58 sec
2013-09-18 00:19:50,262 Stage-2 map = 100%, reduce = 4%, Cumulative CPU 505.58 sec
2013-09-18 00:19:51,267 Stage-2 map = 100%, reduce = 4%, Cumulative CPU 505.58 sec
2013-09-18 00:19:52,273 Stage-2 map = 100%, reduce = 4%, Cumulative CPU 505.58 sec
2013-09-18 00:19:53,277 Stage-2 map = 100%, reduce = 4%, Cumulative CPU 505.58 sec
2013-09-18 00:19:54,281 Stage-2 map = 100%, reduce = 4%, Cumulative CPU 505.58 sec
2013-09-18 00:19:55,285 Stage-2 map = 100%, reduce = 19%, Cumulative CPU 505.58 sec
2013-09-18 00:19:56,289 Stage-2 map = 100%, reduce = 19%, Cumulative CPU 505.58 sec
2013-09-18 00:19:57,293 Stage-2 map = 100%, reduce = 19%, Cumulative CPU 505.58 sec
2013-09-18 00:19:58,296 Stage-2 map = 100%, reduce = 19%, Cumulative CPU 505.58 sec
2013-09-18 00:19:59,300 Stage-2 map = 100%, reduce = 19%, Cumulative CPU 505.58 sec
2013-09-18 00:20:00,305 Stage-2 map = 100%, reduce = 19%, Cumulative CPU 505.58 sec
2013-09-18 00:20:01,309 Stage-2 map = 100%, reduce = 22%, Cumulative CPU 505.58 sec
2013-09-18 00:20:02,313 Stage-2 map = 100%, reduce = 22%, Cumulative CPU 505.58 sec
2013-09-18 00:20:03,317 Stage-2 map = 100%, reduce = 22%, Cumulative CPU 505.58 sec
2013-09-18 00:20:04,322 Stage-2 map = 100%, reduce = 26%, Cumulative CPU 505.58 sec
2013-09-18 00:20:05,326 Stage-2 map = 100%, reduce = 26%, Cumulative CPU 505.58 sec
2013-09-18 00:20:06,329 Stage-2 map = 100%, reduce = 26%, Cumulative CPU 505.58 sec
2013-09-18 00:20:07,334 Stage-2 map = 100%, reduce = 30%, Cumulative CPU 505.58 sec
2013-09-18 00:20:08,338 Stage-2 map = 100%, reduce = 30%, Cumulative CPU 505.58 sec
2013-09-18 00:20:09,342 Stage-2 map = 100%, reduce = 30%, Cumulative CPU 505.58 sec
2013-09-18 00:20:10,347 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 505.58 sec
2013-09-18 00:20:11,357 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 505.58 sec
2013-09-18 00:20:12,362 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 505.58 sec
2013-09-18 00:20:13,366 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 505.58 sec
2013-09-18 00:20:14,371 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 505.58 sec
2013-09-18 00:20:15,376 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 505.58 sec
2013-09-18 00:20:16,381 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 505.58 sec
2013-09-18 00:20:17,386 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 505.58 sec
2013-09-18 00:20:18,391 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 505.58 sec
2013-09-18 00:20:19,396 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 505.58 sec
2013-09-18 00:20:20,401 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 505.58 sec
2013-09-18 00:20:21,405 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 505.58 sec
2013-09-18 00:20:22,410 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 505.58 sec
2013-09-18 00:20:24,399 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 505.58 sec
2013-09-18 00:20:25,403 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 505.58 sec
2013-09-18 00:20:26,407 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 505.58 sec
2013-09-18 00:20:27,761 Stage-2 map = 100%, reduce = 67%, Cumulative CPU 505.58 sec
2013-09-18 00:20:28,766 Stage-2 map = 100%, reduce = 67%, Cumulative CPU 505.58 sec
2013-09-18 00:20:29,947 Stage-2 map = 100%, reduce = 67%, Cumulative CPU 505.58 sec
2013-09-18 00:20:30,952 Stage-2 map = 100%, reduce = 67%, Cumulative CPU 505.58 sec
2013-09-18 00:20:31,956 Stage-2 map = 100%, reduce = 67%, Cumulative CPU 505.58 sec
2013-09-18 00:20:32,961 Stage-2 map = 100%, reduce = 67%, Cumulative CPU 505.58 sec
2013-09-18 00:20:33,966 Stage-2 map = 100%, reduce = 71%, Cumulative CPU 505.58 sec
2013-09-18 00:20:34,970 Stage-2 map = 100%, reduce = 71%, Cumulative CPU 544.29 sec
2013-09-18 00:20:35,975 Stage-2 map = 100%, reduce = 71%, Cumulative CPU 544.29 sec
2013-09-18 00:20:36,979 Stage-2 map = 100%, reduce = 76%, Cumulative CPU 544.29 sec
2013-09-18 00:20:37,984 Stage-2 map = 100%, reduce = 76%, Cumulative CPU 544.29 sec
2013-09-18 00:20:38,988 Stage-2 map = 100%, reduce = 76%, Cumulative CPU 544.29 sec
2013-09-18 00:20:39,999 Stage-2 map = 100%, reduce = 76%, Cumulative CPU 544.29 sec
2013-09-18 00:20:41,004 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 555.06 sec
2013-09-18 00:20:42,008 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 555.06 sec
MapReduce Total cumulative CPU time: 9 minutes 15 seconds 60 msec
Ended Job = job_201309172235_0093
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 11 Cumulative CPU: 1786.03 sec HDFS Read: 843455069 HDFS Write: 2238428518 SUCCESS
Job 1: Map: 9 Reduce: 1 Cumulative CPU: 555.06 sec HDFS Read: 2238513761 HDFS Write: 278 SUCCESS
Total MapReduce CPU Time Spent: 39 minutes 1 seconds 90 msec
OK
Time taken: 234.271 seconds, Fetched: 10 row(s)
hive> quit;
-- ещё более сложная агрегация, не стоит выполнять на больших таблицах.;
times: 1
query: SELECT UserID FROM hits_100m WHERE UserID = 12345678901234567890;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_7682@mturlrep13_201309180020_537514801.txt
hive> SELECT UserID FROM hits_100m WHERE UserID = 12345678901234567890;;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0094
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 0
2013-09-18 00:20:59,984 Stage-1 map = 0%, reduce = 0%
2013-09-18 00:21:09,101 Stage-1 map = 5%, reduce = 0%
2013-09-18 00:21:10,107 Stage-1 map = 16%, reduce = 0%
2013-09-18 00:21:14,125 Stage-1 map = 32%, reduce = 0%
2013-09-18 00:21:15,130 Stage-1 map = 34%, reduce = 0%
2013-09-18 00:21:17,153 Stage-1 map = 35%, reduce = 0%, Cumulative CPU 4.53 sec
2013-09-18 00:21:19,500 Stage-1 map = 64%, reduce = 0%, Cumulative CPU 15.48 sec
2013-09-18 00:21:20,507 Stage-1 map = 67%, reduce = 0%, Cumulative CPU 35.24 sec
2013-09-18 00:21:21,514 Stage-1 map = 93%, reduce = 0%, Cumulative CPU 170.31 sec
2013-09-18 00:21:22,521 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 199.34 sec
2013-09-18 00:21:23,526 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 199.34 sec
MapReduce Total cumulative CPU time: 3 minutes 19 seconds 340 msec
Ended Job = job_201309172235_0094
MapReduce Jobs Launched:
Job 0: Map: 37 Cumulative CPU: 199.34 sec HDFS Read: 663537300 HDFS Write: 0 SUCCESS
Total MapReduce CPU Time Spent: 3 minutes 19 seconds 340 msec
OK
Time taken: 34.019 seconds
hive> quit;
times: 2
query: SELECT UserID FROM hits_100m WHERE UserID = 12345678901234567890;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_12482@mturlrep13_201309180021_1171483805.txt
hive> SELECT UserID FROM hits_100m WHERE UserID = 12345678901234567890;;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0095
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 0
2013-09-18 00:21:32,957 Stage-1 map = 0%, reduce = 0%
2013-09-18 00:21:43,193 Stage-1 map = 36%, reduce = 0%, Cumulative CPU 9.02 sec
2013-09-18 00:21:44,200 Stage-1 map = 92%, reduce = 0%, Cumulative CPU 148.9 sec
2013-09-18 00:21:45,208 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 193.53 sec
2013-09-18 00:21:46,214 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 193.53 sec
2013-09-18 00:21:47,220 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 193.53 sec
MapReduce Total cumulative CPU time: 3 minutes 13 seconds 530 msec
Ended Job = job_201309172235_0095
MapReduce Jobs Launched:
Job 0: Map: 37 Cumulative CPU: 193.53 sec HDFS Read: 663537300 HDFS Write: 0 SUCCESS
Total MapReduce CPU Time Spent: 3 minutes 13 seconds 530 msec
OK
Time taken: 21.834 seconds
hive> quit;
times: 3
query: SELECT UserID FROM hits_100m WHERE UserID = 12345678901234567890;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_17377@mturlrep13_201309180021_991857587.txt
hive> SELECT UserID FROM hits_100m WHERE UserID = 12345678901234567890;;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0096
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 0
2013-09-18 00:21:56,800 Stage-1 map = 0%, reduce = 0%
2013-09-18 00:22:07,146 Stage-1 map = 26%, reduce = 0%, Cumulative CPU 4.76 sec
2013-09-18 00:22:08,183 Stage-1 map = 99%, reduce = 0%, Cumulative CPU 190.17 sec
2013-09-18 00:22:09,190 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 196.69 sec
2013-09-18 00:22:10,195 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 196.69 sec
MapReduce Total cumulative CPU time: 3 minutes 16 seconds 690 msec
Ended Job = job_201309172235_0096
MapReduce Jobs Launched:
Job 0: Map: 37 Cumulative CPU: 196.69 sec HDFS Read: 663537300 HDFS Write: 0 SUCCESS
Total MapReduce CPU Time Spent: 3 minutes 16 seconds 690 msec
OK
Time taken: 21.08 seconds
hive> quit;
-- мощная фильтрация по столбцу типа UInt64.;
times: 1
query: SELECT count(*) AS c FROM hits_100m WHERE URL LIKE '%metrika%';
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_22377@mturlrep13_201309180022_1306519242.txt
hive> SELECT count(*) AS c FROM hits_100m WHERE URL LIKE '%metrika%';;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0097
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 1
2013-09-18 00:22:28,083 Stage-1 map = 0%, reduce = 0%
2013-09-18 00:22:37,160 Stage-1 map = 1%, reduce = 0%
2013-09-18 00:22:38,170 Stage-1 map = 6%, reduce = 0%
2013-09-18 00:22:42,194 Stage-1 map = 20%, reduce = 0%
2013-09-18 00:22:46,226 Stage-1 map = 46%, reduce = 0%
2013-09-18 00:22:47,241 Stage-1 map = 47%, reduce = 0%, Cumulative CPU 10.63 sec
2013-09-18 00:22:48,259 Stage-1 map = 50%, reduce = 0%, Cumulative CPU 28.62 sec
2013-09-18 00:22:49,271 Stage-1 map = 68%, reduce = 0%, Cumulative CPU 38.64 sec
2013-09-18 00:22:50,278 Stage-1 map = 71%, reduce = 0%, Cumulative CPU 61.43 sec
2013-09-18 00:22:51,542 Stage-1 map = 82%, reduce = 0%, Cumulative CPU 244.69 sec
2013-09-18 00:22:52,548 Stage-1 map = 96%, reduce = 0%, Cumulative CPU 321.75 sec
2013-09-18 00:22:53,555 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 417.5 sec
2013-09-18 00:22:54,560 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 417.5 sec
2013-09-18 00:22:55,566 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 417.5 sec
2013-09-18 00:22:56,572 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 417.5 sec
2013-09-18 00:22:57,581 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 420.38 sec
2013-09-18 00:22:58,586 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 420.38 sec
MapReduce Total cumulative CPU time: 7 minutes 0 seconds 380 msec
Ended Job = job_201309172235_0097
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 1 Cumulative CPU: 420.38 sec HDFS Read: 882764316 HDFS Write: 5 SUCCESS
Total MapReduce CPU Time Spent: 7 minutes 0 seconds 380 msec
OK
5238
Time taken: 41.195 seconds, Fetched: 1 row(s)
hive> quit;
times: 2
query: SELECT count(*) AS c FROM hits_100m WHERE URL LIKE '%metrika%';
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_27348@mturlrep13_201309180023_143827785.txt
hive> SELECT count(*) AS c FROM hits_100m WHERE URL LIKE '%metrika%';;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0098
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 1
2013-09-18 00:23:09,675 Stage-1 map = 0%, reduce = 0%
2013-09-18 00:23:19,749 Stage-1 map = 10%, reduce = 0%
2013-09-18 00:23:20,754 Stage-1 map = 12%, reduce = 0%
2013-09-18 00:23:21,760 Stage-1 map = 13%, reduce = 0%
2013-09-18 00:23:22,765 Stage-1 map = 30%, reduce = 0%
2013-09-18 00:23:23,779 Stage-1 map = 43%, reduce = 0%, Cumulative CPU 8.29 sec
2013-09-18 00:23:24,790 Stage-1 map = 47%, reduce = 0%, Cumulative CPU 8.29 sec
2013-09-18 00:23:25,805 Stage-1 map = 57%, reduce = 0%, Cumulative CPU 38.21 sec
2013-09-18 00:23:26,811 Stage-1 map = 74%, reduce = 0%, Cumulative CPU 125.77 sec
2013-09-18 00:23:27,819 Stage-1 map = 88%, reduce = 0%, Cumulative CPU 297.36 sec
2013-09-18 00:23:28,825 Stage-1 map = 99%, reduce = 0%, Cumulative CPU 425.9 sec
2013-09-18 00:23:29,830 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 439.17 sec
2013-09-18 00:23:30,836 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 439.17 sec
2013-09-18 00:23:31,841 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 439.17 sec
2013-09-18 00:23:32,847 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 439.17 sec
2013-09-18 00:23:33,853 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 439.17 sec
2013-09-18 00:23:34,858 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 439.17 sec
2013-09-18 00:23:35,865 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 441.94 sec
2013-09-18 00:23:36,871 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 441.94 sec
MapReduce Total cumulative CPU time: 7 minutes 21 seconds 940 msec
Ended Job = job_201309172235_0098
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 1 Cumulative CPU: 441.94 sec HDFS Read: 882764316 HDFS Write: 5 SUCCESS
Total MapReduce CPU Time Spent: 7 minutes 21 seconds 940 msec
OK
5238
Time taken: 36.443 seconds, Fetched: 1 row(s)
hive> quit;
times: 3
query: SELECT count(*) AS c FROM hits_100m WHERE URL LIKE '%metrika%';
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_32317@mturlrep13_201309180023_1607167986.txt
hive> SELECT count(*) AS c FROM hits_100m WHERE URL LIKE '%metrika%';;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0099
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 1
2013-09-18 00:23:47,464 Stage-1 map = 0%, reduce = 0%
2013-09-18 00:23:56,520 Stage-1 map = 3%, reduce = 0%
2013-09-18 00:23:57,526 Stage-1 map = 11%, reduce = 0%
2013-09-18 00:23:59,535 Stage-1 map = 15%, reduce = 0%
2013-09-18 00:24:00,541 Stage-1 map = 38%, reduce = 0%
2013-09-18 00:24:01,559 Stage-1 map = 46%, reduce = 0%, Cumulative CPU 8.66 sec
2013-09-18 00:24:02,572 Stage-1 map = 49%, reduce = 0%, Cumulative CPU 8.66 sec
2013-09-18 00:24:03,580 Stage-1 map = 64%, reduce = 0%, Cumulative CPU 50.46 sec
2013-09-18 00:24:04,591 Stage-1 map = 76%, reduce = 0%, Cumulative CPU 152.61 sec
2013-09-18 00:24:05,597 Stage-1 map = 93%, reduce = 0%, Cumulative CPU 361.23 sec
2013-09-18 00:24:06,603 Stage-1 map = 98%, reduce = 0%, Cumulative CPU 427.75 sec
2013-09-18 00:24:07,609 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 441.29 sec
2013-09-18 00:24:08,613 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 441.29 sec
2013-09-18 00:24:09,618 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 441.29 sec
2013-09-18 00:24:10,624 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 441.29 sec
2013-09-18 00:24:11,696 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 441.29 sec
2013-09-18 00:24:12,703 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 444.08 sec
2013-09-18 00:24:13,709 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 444.08 sec
2013-09-18 00:24:14,715 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 444.08 sec
MapReduce Total cumulative CPU time: 7 minutes 24 seconds 80 msec
Ended Job = job_201309172235_0099
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 1 Cumulative CPU: 444.08 sec HDFS Read: 882764316 HDFS Write: 5 SUCCESS
Total MapReduce CPU Time Spent: 7 minutes 24 seconds 80 msec
OK
5238
Time taken: 35.979 seconds, Fetched: 1 row(s)
hive> quit;
-- фильтрация по поиску подстроки в строке.;
times: 1
query: SELECT SearchPhrase, MAX(URL), count(*) AS c FROM hits_100m WHERE URL LIKE '%metrika%' AND SearchPhrase != '' GROUP BY SearchPhrase ORDER BY c DESC LIMIT 10;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_4931@mturlrep13_201309180024_377998774.txt
hive> SELECT SearchPhrase, MAX(URL), count(*) AS c FROM hits_100m WHERE URL LIKE '%metrika%' AND SearchPhrase != '' GROUP BY SearchPhrase ORDER BY c DESC LIMIT 10;;
Total MapReduce jobs = 2
Launching Job 1 out of 2
Number of reduce tasks not specified. Estimated from input data size: 11
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0100
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 11
2013-09-18 00:24:32,532 Stage-1 map = 0%, reduce = 0%
2013-09-18 00:24:41,635 Stage-1 map = 1%, reduce = 0%
2013-09-18 00:24:42,650 Stage-1 map = 4%, reduce = 0%
2013-09-18 00:24:46,694 Stage-1 map = 15%, reduce = 0%
2013-09-18 00:24:49,790 Stage-1 map = 39%, reduce = 0%
2013-09-18 00:24:50,805 Stage-1 map = 40%, reduce = 0%, Cumulative CPU 12.57 sec
2013-09-18 00:24:54,895 Stage-1 map = 59%, reduce = 0%, Cumulative CPU 60.14 sec
2013-09-18 00:24:55,919 Stage-1 map = 77%, reduce = 0%, Cumulative CPU 143.58 sec
2013-09-18 00:24:56,938 Stage-1 map = 81%, reduce = 0%, Cumulative CPU 219.04 sec
2013-09-18 00:24:57,944 Stage-1 map = 95%, reduce = 0%, Cumulative CPU 404.21 sec
2013-09-18 00:24:58,950 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 468.52 sec
2013-09-18 00:24:59,956 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 468.52 sec
2013-09-18 00:25:00,962 Stage-1 map = 100%, reduce = 3%, Cumulative CPU 468.52 sec
2013-09-18 00:25:01,971 Stage-1 map = 100%, reduce = 18%, Cumulative CPU 471.46 sec
2013-09-18 00:25:02,978 Stage-1 map = 100%, reduce = 39%, Cumulative CPU 474.57 sec
2013-09-18 00:25:03,988 Stage-1 map = 100%, reduce = 73%, Cumulative CPU 490.52 sec
2013-09-18 00:25:04,995 Stage-1 map = 100%, reduce = 91%, Cumulative CPU 500.12 sec
2013-09-18 00:25:06,001 Stage-1 map = 100%, reduce = 94%, Cumulative CPU 500.12 sec
2013-09-18 00:25:07,008 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 503.18 sec
2013-09-18 00:25:08,019 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 503.18 sec
MapReduce Total cumulative CPU time: 8 minutes 23 seconds 180 msec
Ended Job = job_201309172235_0100
Launching Job 2 out of 2
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0101
Hadoop job information for Stage-2: number of mappers: 1; number of reducers: 1
2013-09-18 00:25:11,520 Stage-2 map = 0%, reduce = 0%
2013-09-18 00:25:13,528 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.73 sec
2013-09-18 00:25:14,534 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.73 sec
2013-09-18 00:25:15,540 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.73 sec
2013-09-18 00:25:16,545 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.73 sec
2013-09-18 00:25:17,549 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.73 sec
2013-09-18 00:25:18,554 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.73 sec
2013-09-18 00:25:19,559 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.73 sec
2013-09-18 00:25:20,564 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 0.73 sec
2013-09-18 00:25:21,569 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 2.17 sec
2013-09-18 00:25:22,574 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 2.17 sec
MapReduce Total cumulative CPU time: 2 seconds 170 msec
Ended Job = job_201309172235_0101
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 11 Cumulative CPU: 503.18 sec HDFS Read: 1060099081 HDFS Write: 11615 SUCCESS
Job 1: Map: 1 Reduce: 1 Cumulative CPU: 2.17 sec HDFS Read: 15192 HDFS Write: 1152 SUCCESS
Total MapReduce CPU Time Spent: 8 minutes 25 seconds 350 msec
OK
Time taken: 60.385 seconds, Fetched: 10 row(s)
hive> quit;
times: 2
query: SELECT SearchPhrase, MAX(URL), count(*) AS c FROM hits_100m WHERE URL LIKE '%metrika%' AND SearchPhrase != '' GROUP BY SearchPhrase ORDER BY c DESC LIMIT 10;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_12874@mturlrep13_201309180025_385714226.txt
hive> SELECT SearchPhrase, MAX(URL), count(*) AS c FROM hits_100m WHERE URL LIKE '%metrika%' AND SearchPhrase != '' GROUP BY SearchPhrase ORDER BY c DESC LIMIT 10;;
Total MapReduce jobs = 2
Launching Job 1 out of 2
Number of reduce tasks not specified. Estimated from input data size: 11
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0102
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 11
2013-09-18 00:25:33,396 Stage-1 map = 0%, reduce = 0%
2013-09-18 00:25:43,564 Stage-1 map = 9%, reduce = 0%
2013-09-18 00:25:45,601 Stage-1 map = 10%, reduce = 0%
2013-09-18 00:25:46,613 Stage-1 map = 33%, reduce = 0%
2013-09-18 00:25:47,620 Stage-1 map = 37%, reduce = 0%
2013-09-18 00:25:48,634 Stage-1 map = 43%, reduce = 0%, Cumulative CPU 9.38 sec
2013-09-18 00:25:49,642 Stage-1 map = 49%, reduce = 0%, Cumulative CPU 9.38 sec
2013-09-18 00:25:50,658 Stage-1 map = 57%, reduce = 0%, Cumulative CPU 9.38 sec
2013-09-18 00:25:51,664 Stage-1 map = 65%, reduce = 0%, Cumulative CPU 21.89 sec
2013-09-18 00:25:52,694 Stage-1 map = 82%, reduce = 0%, Cumulative CPU 218.15 sec
2013-09-18 00:25:53,701 Stage-1 map = 99%, reduce = 0%, Cumulative CPU 479.2 sec
2013-09-18 00:25:54,708 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 493.05 sec
2013-09-18 00:25:55,715 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 493.05 sec
2013-09-18 00:25:56,721 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 493.05 sec
2013-09-18 00:25:57,727 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 493.05 sec
2013-09-18 00:25:58,733 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 493.05 sec
2013-09-18 00:25:59,740 Stage-1 map = 100%, reduce = 3%, Cumulative CPU 493.05 sec
2013-09-18 00:26:00,748 Stage-1 map = 100%, reduce = 21%, Cumulative CPU 493.05 sec
2013-09-18 00:26:01,758 Stage-1 map = 100%, reduce = 61%, Cumulative CPU 510.23 sec
2013-09-18 00:26:02,768 Stage-1 map = 100%, reduce = 88%, Cumulative CPU 524.01 sec
2013-09-18 00:26:03,775 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 530.35 sec
2013-09-18 00:26:04,781 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 530.35 sec
MapReduce Total cumulative CPU time: 8 minutes 50 seconds 350 msec
Ended Job = job_201309172235_0102
Launching Job 2 out of 2
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0103
Hadoop job information for Stage-2: number of mappers: 1; number of reducers: 1
2013-09-18 00:26:08,247 Stage-2 map = 0%, reduce = 0%
2013-09-18 00:26:09,253 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.71 sec
2013-09-18 00:26:10,258 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.71 sec
2013-09-18 00:26:11,263 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.71 sec
2013-09-18 00:26:12,269 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.71 sec
2013-09-18 00:26:13,275 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.71 sec
2013-09-18 00:26:14,280 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.71 sec
2013-09-18 00:26:15,284 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.71 sec
2013-09-18 00:26:16,289 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.71 sec
2013-09-18 00:26:17,294 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 2.35 sec
2013-09-18 00:26:18,299 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 2.35 sec
2013-09-18 00:26:19,304 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 2.35 sec
MapReduce Total cumulative CPU time: 2 seconds 350 msec
Ended Job = job_201309172235_0103
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 11 Cumulative CPU: 530.35 sec HDFS Read: 1060099081 HDFS Write: 11615 SUCCESS
Job 1: Map: 1 Reduce: 1 Cumulative CPU: 2.35 sec HDFS Read: 15192 HDFS Write: 1152 SUCCESS
Total MapReduce CPU Time Spent: 8 minutes 52 seconds 700 msec
OK
Time taken: 54.888 seconds, Fetched: 10 row(s)
hive> quit;
times: 3
query: SELECT SearchPhrase, MAX(URL), count(*) AS c FROM hits_100m WHERE URL LIKE '%metrika%' AND SearchPhrase != '' GROUP BY SearchPhrase ORDER BY c DESC LIMIT 10;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_19768@mturlrep13_201309180026_1068991002.txt
hive> SELECT SearchPhrase, MAX(URL), count(*) AS c FROM hits_100m WHERE URL LIKE '%metrika%' AND SearchPhrase != '' GROUP BY SearchPhrase ORDER BY c DESC LIMIT 10;;
Total MapReduce jobs = 2
Launching Job 1 out of 2
Number of reduce tasks not specified. Estimated from input data size: 11
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0104
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 11
2013-09-18 00:26:30,060 Stage-1 map = 0%, reduce = 0%
2013-09-18 00:26:39,192 Stage-1 map = 1%, reduce = 0%
2013-09-18 00:26:40,200 Stage-1 map = 9%, reduce = 0%
2013-09-18 00:26:41,206 Stage-1 map = 10%, reduce = 0%
2013-09-18 00:26:42,212 Stage-1 map = 11%, reduce = 0%
2013-09-18 00:26:43,218 Stage-1 map = 29%, reduce = 0%
2013-09-18 00:26:44,224 Stage-1 map = 35%, reduce = 0%
2013-09-18 00:26:45,239 Stage-1 map = 40%, reduce = 0%, Cumulative CPU 9.36 sec
2013-09-18 00:26:46,268 Stage-1 map = 45%, reduce = 0%, Cumulative CPU 9.36 sec
2013-09-18 00:26:47,275 Stage-1 map = 52%, reduce = 0%, Cumulative CPU 9.36 sec
2013-09-18 00:26:48,316 Stage-1 map = 60%, reduce = 0%, Cumulative CPU 21.78 sec
2013-09-18 00:26:49,334 Stage-1 map = 78%, reduce = 0%, Cumulative CPU 217.46 sec
2013-09-18 00:26:50,341 Stage-1 map = 94%, reduce = 0%, Cumulative CPU 406.39 sec
2013-09-18 00:26:51,348 Stage-1 map = 99%, reduce = 0%, Cumulative CPU 477.62 sec
2013-09-18 00:26:52,356 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 491.94 sec
2013-09-18 00:26:53,362 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 491.94 sec
2013-09-18 00:26:54,368 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 491.94 sec
2013-09-18 00:26:55,374 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 491.94 sec
2013-09-18 00:26:56,380 Stage-1 map = 100%, reduce = 3%, Cumulative CPU 491.94 sec
2013-09-18 00:26:57,388 Stage-1 map = 100%, reduce = 12%, Cumulative CPU 491.94 sec
2013-09-18 00:26:58,397 Stage-1 map = 100%, reduce = 52%, Cumulative CPU 504.87 sec
2013-09-18 00:26:59,404 Stage-1 map = 100%, reduce = 82%, Cumulative CPU 518.69 sec
2013-09-18 00:27:00,410 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 528.53 sec
2013-09-18 00:27:01,416 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 528.53 sec
2013-09-18 00:27:02,423 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 528.53 sec
MapReduce Total cumulative CPU time: 8 minutes 48 seconds 530 msec
Ended Job = job_201309172235_0104
Launching Job 2 out of 2
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0105
Hadoop job information for Stage-2: number of mappers: 1; number of reducers: 1
2013-09-18 00:27:06,499 Stage-2 map = 0%, reduce = 0%
2013-09-18 00:27:07,505 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.72 sec
2013-09-18 00:27:08,522 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.72 sec
2013-09-18 00:27:09,527 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.72 sec
2013-09-18 00:27:10,532 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.72 sec
2013-09-18 00:27:11,538 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.72 sec
2013-09-18 00:27:12,543 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.72 sec
2013-09-18 00:27:13,547 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.72 sec
2013-09-18 00:27:14,552 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.72 sec
2013-09-18 00:27:15,558 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 2.26 sec
2013-09-18 00:27:16,563 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 2.26 sec
2013-09-18 00:27:17,568 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 2.26 sec
MapReduce Total cumulative CPU time: 2 seconds 260 msec
Ended Job = job_201309172235_0105
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 11 Cumulative CPU: 528.53 sec HDFS Read: 1060099081 HDFS Write: 11615 SUCCESS
Job 1: Map: 1 Reduce: 1 Cumulative CPU: 2.26 sec HDFS Read: 15192 HDFS Write: 1152 SUCCESS
Total MapReduce CPU Time Spent: 8 minutes 50 seconds 790 msec
OK
Time taken: 56.541 seconds, Fetched: 10 row(s)
hive> quit;
-- вынимаем большие столбцы, фильтрация по строке.;
times: 1
query: SELECT SearchPhrase, MAX(URL), MAX(Title), count(*) AS c, count(DISTINCT UserID) FROM hits_100m WHERE Title LIKE '%Яндекс%' AND URL NOT LIKE '%.yandex.%' AND SearchPhrase != '' GROUP BY SearchPhrase ORDER BY c DESC LIMIT 10;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_26625@mturlrep13_201309180027_303480636.txt
hive> SELECT SearchPhrase, MAX(URL), MAX(Title), count(*) AS c, count(DISTINCT UserID) FROM hits_100m WHERE Title LIKE '%Яндекс%' AND URL NOT LIKE '%.yandex.%' AND SearchPhrase != '' GROUP BY SearchPhrase ORDER BY c DESC LIMIT 10;;
Total MapReduce jobs = 2
Launching Job 1 out of 2
Number of reduce tasks not specified. Estimated from input data size: 11
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0106
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 11
2013-09-18 00:27:35,507 Stage-1 map = 0%, reduce = 0%
2013-09-18 00:27:45,586 Stage-1 map = 2%, reduce = 0%
2013-09-18 00:27:48,661 Stage-1 map = 10%, reduce = 0%
2013-09-18 00:27:50,957 Stage-1 map = 14%, reduce = 0%
2013-09-18 00:27:51,973 Stage-1 map = 20%, reduce = 0%
2013-09-18 00:27:53,983 Stage-1 map = 22%, reduce = 0%
2013-09-18 00:27:56,028 Stage-1 map = 35%, reduce = 0%
2013-09-18 00:27:58,127 Stage-1 map = 40%, reduce = 0%, Cumulative CPU 12.89 sec
2013-09-18 00:27:59,135 Stage-1 map = 47%, reduce = 0%, Cumulative CPU 12.89 sec
2013-09-18 00:28:00,141 Stage-1 map = 50%, reduce = 0%, Cumulative CPU 27.18 sec
2013-09-18 00:28:01,159 Stage-1 map = 52%, reduce = 0%, Cumulative CPU 27.18 sec
2013-09-18 00:28:02,167 Stage-1 map = 63%, reduce = 0%, Cumulative CPU 59.55 sec
2013-09-18 00:28:04,208 Stage-1 map = 67%, reduce = 0%, Cumulative CPU 92.71 sec
2013-09-18 00:28:05,214 Stage-1 map = 79%, reduce = 0%, Cumulative CPU 175.04 sec
2013-09-18 00:28:06,329 Stage-1 map = 85%, reduce = 0%, Cumulative CPU 341.42 sec
2013-09-18 00:28:07,336 Stage-1 map = 97%, reduce = 0%, Cumulative CPU 556.8 sec
2013-09-18 00:28:08,342 Stage-1 map = 100%, reduce = 2%, Cumulative CPU 608.33 sec
2013-09-18 00:28:09,348 Stage-1 map = 100%, reduce = 2%, Cumulative CPU 608.33 sec
2013-09-18 00:28:10,354 Stage-1 map = 100%, reduce = 8%, Cumulative CPU 608.33 sec
2013-09-18 00:28:11,362 Stage-1 map = 100%, reduce = 20%, Cumulative CPU 611.74 sec
2013-09-18 00:28:12,369 Stage-1 map = 100%, reduce = 47%, Cumulative CPU 622.43 sec
2013-09-18 00:28:13,376 Stage-1 map = 100%, reduce = 69%, Cumulative CPU 632.58 sec
2013-09-18 00:28:14,383 Stage-1 map = 100%, reduce = 75%, Cumulative CPU 632.58 sec
2013-09-18 00:28:15,389 Stage-1 map = 100%, reduce = 94%, Cumulative CPU 643.54 sec
2013-09-18 00:28:16,396 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 647.01 sec
2013-09-18 00:28:17,401 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 647.01 sec
MapReduce Total cumulative CPU time: 10 minutes 47 seconds 10 msec
Ended Job = job_201309172235_0106
Launching Job 2 out of 2
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0107
Hadoop job information for Stage-2: number of mappers: 1; number of reducers: 1
2013-09-18 00:28:19,874 Stage-2 map = 0%, reduce = 0%
2013-09-18 00:28:21,883 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 1.61 sec
2013-09-18 00:28:22,887 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 1.61 sec
2013-09-18 00:28:23,892 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 1.61 sec
2013-09-18 00:28:24,896 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 1.61 sec
2013-09-18 00:28:25,900 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 1.61 sec
2013-09-18 00:28:26,904 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 1.61 sec
2013-09-18 00:28:27,908 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 1.61 sec
2013-09-18 00:28:28,912 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 1.61 sec
2013-09-18 00:28:29,917 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 1.61 sec
2013-09-18 00:28:30,923 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 3.4 sec
2013-09-18 00:28:31,928 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 3.4 sec
MapReduce Total cumulative CPU time: 3 seconds 400 msec
Ended Job = job_201309172235_0107
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 11 Cumulative CPU: 647.01 sec HDFS Read: 2479854627 HDFS Write: 1092954 SUCCESS
Job 1: Map: 1 Reduce: 1 Cumulative CPU: 3.4 sec HDFS Read: 1096531 HDFS Write: 1307 SUCCESS
Total MapReduce CPU Time Spent: 10 minutes 50 seconds 410 msec
OK
Time taken: 67.257 seconds, Fetched: 10 row(s)
hive> quit;
times: 2
query: SELECT SearchPhrase, MAX(URL), MAX(Title), count(*) AS c, count(DISTINCT UserID) FROM hits_100m WHERE Title LIKE '%Яндекс%' AND URL NOT LIKE '%.yandex.%' AND SearchPhrase != '' GROUP BY SearchPhrase ORDER BY c DESC LIMIT 10;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_1658@mturlrep13_201309180028_324429197.txt
hive> SELECT SearchPhrase, MAX(URL), MAX(Title), count(*) AS c, count(DISTINCT UserID) FROM hits_100m WHERE Title LIKE '%Яндекс%' AND URL NOT LIKE '%.yandex.%' AND SearchPhrase != '' GROUP BY SearchPhrase ORDER BY c DESC LIMIT 10;;
Total MapReduce jobs = 2
Launching Job 1 out of 2
Number of reduce tasks not specified. Estimated from input data size: 11
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0108
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 11
2013-09-18 00:28:42,812 Stage-1 map = 0%, reduce = 0%
2013-09-18 00:28:52,951 Stage-1 map = 1%, reduce = 0%
2013-09-18 00:28:54,963 Stage-1 map = 11%, reduce = 0%
2013-09-18 00:28:55,970 Stage-1 map = 18%, reduce = 0%
2013-09-18 00:28:56,983 Stage-1 map = 19%, reduce = 0%
2013-09-18 00:28:57,990 Stage-1 map = 26%, reduce = 0%
2013-09-18 00:28:59,003 Stage-1 map = 29%, reduce = 0%
2013-09-18 00:29:00,009 Stage-1 map = 32%, reduce = 0%
2013-09-18 00:29:01,027 Stage-1 map = 39%, reduce = 0%
2013-09-18 00:29:02,034 Stage-1 map = 47%, reduce = 0%
2013-09-18 00:29:03,043 Stage-1 map = 51%, reduce = 0%
2013-09-18 00:29:04,091 Stage-1 map = 61%, reduce = 0%, Cumulative CPU 30.86 sec
2013-09-18 00:29:05,114 Stage-1 map = 74%, reduce = 0%, Cumulative CPU 63.18 sec
2013-09-18 00:29:06,133 Stage-1 map = 82%, reduce = 0%, Cumulative CPU 233.25 sec
2013-09-18 00:29:07,141 Stage-1 map = 95%, reduce = 0%, Cumulative CPU 514.05 sec
2013-09-18 00:29:08,148 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 641.77 sec
2013-09-18 00:29:09,154 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 641.77 sec
2013-09-18 00:29:10,160 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 641.77 sec
2013-09-18 00:29:11,166 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 641.77 sec
2013-09-18 00:29:12,173 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 641.77 sec
2013-09-18 00:29:13,179 Stage-1 map = 100%, reduce = 6%, Cumulative CPU 641.77 sec
2013-09-18 00:29:14,186 Stage-1 map = 100%, reduce = 21%, Cumulative CPU 641.77 sec
2013-09-18 00:29:15,211 Stage-1 map = 100%, reduce = 45%, Cumulative CPU 648.85 sec
2013-09-18 00:29:16,218 Stage-1 map = 100%, reduce = 94%, Cumulative CPU 679.72 sec
2013-09-18 00:29:17,224 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 683.49 sec
2013-09-18 00:29:18,230 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 683.49 sec
MapReduce Total cumulative CPU time: 11 minutes 23 seconds 490 msec
Ended Job = job_201309172235_0108
Launching Job 2 out of 2
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0109
Hadoop job information for Stage-2: number of mappers: 1; number of reducers: 1
2013-09-18 00:29:20,736 Stage-2 map = 0%, reduce = 0%
2013-09-18 00:29:23,749 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 1.6 sec
2013-09-18 00:29:24,754 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 1.6 sec
2013-09-18 00:29:25,759 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 1.6 sec
2013-09-18 00:29:26,763 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 1.6 sec
2013-09-18 00:29:27,767 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 1.6 sec
2013-09-18 00:29:28,772 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 1.6 sec
2013-09-18 00:29:29,776 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 1.6 sec
2013-09-18 00:29:30,782 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 1.6 sec
2013-09-18 00:29:31,787 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 3.35 sec
2013-09-18 00:29:32,792 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 3.35 sec
MapReduce Total cumulative CPU time: 3 seconds 350 msec
Ended Job = job_201309172235_0109
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 11 Cumulative CPU: 683.49 sec HDFS Read: 2479854627 HDFS Write: 1092954 SUCCESS
Job 1: Map: 1 Reduce: 1 Cumulative CPU: 3.35 sec HDFS Read: 1096531 HDFS Write: 1307 SUCCESS
Total MapReduce CPU Time Spent: 11 minutes 26 seconds 840 msec
OK
Time taken: 58.995 seconds, Fetched: 10 row(s)
hive> quit;
times: 3
query: SELECT SearchPhrase, MAX(URL), MAX(Title), count(*) AS c, count(DISTINCT UserID) FROM hits_100m WHERE Title LIKE '%Яндекс%' AND URL NOT LIKE '%.yandex.%' AND SearchPhrase != '' GROUP BY SearchPhrase ORDER BY c DESC LIMIT 10;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_9465@mturlrep13_201309180029_1752618536.txt
hive> SELECT SearchPhrase, MAX(URL), MAX(Title), count(*) AS c, count(DISTINCT UserID) FROM hits_100m WHERE Title LIKE '%Яндекс%' AND URL NOT LIKE '%.yandex.%' AND SearchPhrase != '' GROUP BY SearchPhrase ORDER BY c DESC LIMIT 10;;
Total MapReduce jobs = 2
Launching Job 1 out of 2
Number of reduce tasks not specified. Estimated from input data size: 11
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0110
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 11
2013-09-18 00:29:42,551 Stage-1 map = 0%, reduce = 0%
2013-09-18 00:29:53,652 Stage-1 map = 1%, reduce = 0%
2013-09-18 00:29:56,667 Stage-1 map = 16%, reduce = 0%
2013-09-18 00:29:57,673 Stage-1 map = 21%, reduce = 0%
2013-09-18 00:29:58,678 Stage-1 map = 26%, reduce = 0%
2013-09-18 00:29:59,684 Stage-1 map = 28%, reduce = 0%
2013-09-18 00:30:00,690 Stage-1 map = 31%, reduce = 0%
2013-09-18 00:30:01,707 Stage-1 map = 38%, reduce = 0%
2013-09-18 00:30:02,717 Stage-1 map = 43%, reduce = 0%
2013-09-18 00:30:03,733 Stage-1 map = 51%, reduce = 0%
2013-09-18 00:30:04,774 Stage-1 map = 63%, reduce = 0%, Cumulative CPU 14.24 sec
2013-09-18 00:30:05,800 Stage-1 map = 67%, reduce = 0%, Cumulative CPU 45.73 sec
2013-09-18 00:30:06,808 Stage-1 map = 76%, reduce = 0%, Cumulative CPU 128.94 sec
2013-09-18 00:30:07,818 Stage-1 map = 89%, reduce = 0%, Cumulative CPU 335.92 sec
2013-09-18 00:30:08,826 Stage-1 map = 96%, reduce = 0%, Cumulative CPU 549.46 sec
2013-09-18 00:30:09,832 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 641.18 sec
2013-09-18 00:30:10,838 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 641.18 sec
2013-09-18 00:30:11,844 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 641.18 sec
2013-09-18 00:30:12,850 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 641.18 sec
2013-09-18 00:30:13,856 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 641.18 sec
2013-09-18 00:30:14,863 Stage-1 map = 100%, reduce = 15%, Cumulative CPU 641.18 sec
2013-09-18 00:30:15,874 Stage-1 map = 100%, reduce = 36%, Cumulative CPU 644.72 sec
2013-09-18 00:30:16,882 Stage-1 map = 100%, reduce = 76%, Cumulative CPU 667.58 sec
2013-09-18 00:30:17,888 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 682.62 sec
2013-09-18 00:30:18,895 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 682.62 sec
2013-09-18 00:30:19,901 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 682.62 sec
MapReduce Total cumulative CPU time: 11 minutes 22 seconds 620 msec
Ended Job = job_201309172235_0110
Launching Job 2 out of 2
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0111
Hadoop job information for Stage-2: number of mappers: 1; number of reducers: 1
2013-09-18 00:30:23,450 Stage-2 map = 0%, reduce = 0%
2013-09-18 00:30:25,458 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 1.57 sec
2013-09-18 00:30:26,463 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 1.57 sec
2013-09-18 00:30:27,468 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 1.57 sec
2013-09-18 00:30:28,472 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 1.57 sec
2013-09-18 00:30:29,476 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 1.57 sec
2013-09-18 00:30:30,480 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 1.57 sec
2013-09-18 00:30:31,484 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 1.57 sec
2013-09-18 00:30:32,489 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 1.57 sec
2013-09-18 00:30:33,494 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 3.37 sec
2013-09-18 00:30:34,500 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 3.37 sec
MapReduce Total cumulative CPU time: 3 seconds 370 msec
Ended Job = job_201309172235_0111
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 11 Cumulative CPU: 682.62 sec HDFS Read: 2479854627 HDFS Write: 1092954 SUCCESS
Job 1: Map: 1 Reduce: 1 Cumulative CPU: 3.37 sec HDFS Read: 1096531 HDFS Write: 1307 SUCCESS
Total MapReduce CPU Time Spent: 11 minutes 25 seconds 990 msec
OK
Time taken: 59.828 seconds, Fetched: 10 row(s)
hive> quit;
-- чуть больше столбцы.;
times: 1
query: SELECT * FROM hits_100m WHERE URL LIKE '%metrika%' ORDER BY EventTime LIMIT 10;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_17569@mturlrep13_201309180030_564255388.txt
hive> SELECT * FROM hits_100m WHERE URL LIKE '%metrika%' ORDER BY EventTime LIMIT 10;;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0112
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 1
2013-09-18 00:30:52,920 Stage-1 map = 0%, reduce = 0%
2013-09-18 00:31:12,034 Stage-1 map = 1%, reduce = 0%
2013-09-18 00:31:15,535 Stage-1 map = 3%, reduce = 0%
2013-09-18 00:31:16,540 Stage-1 map = 5%, reduce = 0%
2013-09-18 00:31:17,545 Stage-1 map = 7%, reduce = 0%
2013-09-18 00:31:18,551 Stage-1 map = 10%, reduce = 0%
2013-09-18 00:31:19,555 Stage-1 map = 13%, reduce = 0%
2013-09-18 00:31:21,568 Stage-1 map = 15%, reduce = 0%
2013-09-18 00:31:22,572 Stage-1 map = 17%, reduce = 0%
2013-09-18 00:31:23,581 Stage-1 map = 19%, reduce = 0%
2013-09-18 00:31:24,586 Stage-1 map = 20%, reduce = 0%
2013-09-18 00:31:25,609 Stage-1 map = 25%, reduce = 0%
2013-09-18 00:31:26,614 Stage-1 map = 27%, reduce = 0%
2013-09-18 00:31:27,633 Stage-1 map = 28%, reduce = 0%
2013-09-18 00:31:28,641 Stage-1 map = 30%, reduce = 0%
2013-09-18 00:31:29,675 Stage-1 map = 32%, reduce = 0%
2013-09-18 00:31:30,698 Stage-1 map = 33%, reduce = 0%
2013-09-18 00:31:32,058 Stage-1 map = 36%, reduce = 0%
2013-09-18 00:31:33,079 Stage-1 map = 38%, reduce = 0%
2013-09-18 00:31:34,102 Stage-1 map = 40%, reduce = 0%
2013-09-18 00:31:35,117 Stage-1 map = 44%, reduce = 0%
2013-09-18 00:31:36,122 Stage-1 map = 45%, reduce = 0%
2013-09-18 00:31:37,152 Stage-1 map = 46%, reduce = 0%, Cumulative CPU 1131.99 sec
2013-09-18 00:31:38,167 Stage-1 map = 47%, reduce = 0%, Cumulative CPU 1131.99 sec
2013-09-18 00:31:39,186 Stage-1 map = 49%, reduce = 0%, Cumulative CPU 1135.59 sec
2013-09-18 00:31:40,202 Stage-1 map = 51%, reduce = 0%, Cumulative CPU 1135.59 sec
2013-09-18 00:31:41,208 Stage-1 map = 54%, reduce = 0%, Cumulative CPU 1135.59 sec
2013-09-18 00:31:42,221 Stage-1 map = 58%, reduce = 0%, Cumulative CPU 1135.59 sec
2013-09-18 00:31:43,226 Stage-1 map = 60%, reduce = 0%, Cumulative CPU 1135.59 sec
2013-09-18 00:31:44,286 Stage-1 map = 61%, reduce = 0%, Cumulative CPU 1135.59 sec
2013-09-18 00:31:45,292 Stage-1 map = 63%, reduce = 0%, Cumulative CPU 1135.59 sec
2013-09-18 00:31:46,298 Stage-1 map = 64%, reduce = 0%, Cumulative CPU 1135.59 sec
2013-09-18 00:31:47,305 Stage-1 map = 66%, reduce = 0%, Cumulative CPU 1135.59 sec
2013-09-18 00:31:48,325 Stage-1 map = 69%, reduce = 0%, Cumulative CPU 1135.59 sec
2013-09-18 00:31:49,332 Stage-1 map = 73%, reduce = 0%, Cumulative CPU 1135.59 sec
2013-09-18 00:31:50,340 Stage-1 map = 75%, reduce = 0%, Cumulative CPU 1135.59 sec
2013-09-18 00:31:51,346 Stage-1 map = 76%, reduce = 0%, Cumulative CPU 1150.66 sec
2013-09-18 00:31:52,357 Stage-1 map = 78%, reduce = 0%, Cumulative CPU 1150.66 sec
2013-09-18 00:31:53,363 Stage-1 map = 80%, reduce = 0%, Cumulative CPU 1201.45 sec
2013-09-18 00:31:54,370 Stage-1 map = 85%, reduce = 0%, Cumulative CPU 1286.34 sec
2013-09-18 00:31:55,375 Stage-1 map = 88%, reduce = 0%, Cumulative CPU 1340.63 sec
2013-09-18 00:31:56,384 Stage-1 map = 91%, reduce = 0%, Cumulative CPU 1412.26 sec
2013-09-18 00:31:57,396 Stage-1 map = 94%, reduce = 0%, Cumulative CPU 1529.38 sec
2013-09-18 00:31:58,401 Stage-1 map = 97%, reduce = 0%, Cumulative CPU 1690.25 sec
2013-09-18 00:31:59,406 Stage-1 map = 99%, reduce = 0%, Cumulative CPU 1753.24 sec
2013-09-18 00:32:00,412 Stage-1 map = 99%, reduce = 0%, Cumulative CPU 1777.14 sec
2013-09-18 00:32:01,416 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1828.2 sec
2013-09-18 00:32:02,421 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1828.2 sec
2013-09-18 00:32:03,426 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1828.2 sec
2013-09-18 00:32:04,430 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1828.2 sec
2013-09-18 00:32:05,435 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1828.2 sec
2013-09-18 00:32:06,439 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1828.2 sec
2013-09-18 00:32:07,444 Stage-1 map = 100%, reduce = 67%, Cumulative CPU 1828.2 sec
2013-09-18 00:32:08,451 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 1831.77 sec
2013-09-18 00:32:09,457 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 1831.77 sec
MapReduce Total cumulative CPU time: 30 minutes 31 seconds 770 msec
Ended Job = job_201309172235_0112
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 1 Cumulative CPU: 1831.77 sec HDFS Read: 10081441390 HDFS Write: 5802 SUCCESS
Total MapReduce CPU Time Spent: 30 minutes 31 seconds 770 msec
OK
Time taken: 87.697 seconds, Fetched: 10 row(s)
hive> quit;
times: 2
query: SELECT * FROM hits_100m WHERE URL LIKE '%metrika%' ORDER BY EventTime LIMIT 10;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_22644@mturlrep13_201309180032_2112140807.txt
hive> SELECT * FROM hits_100m WHERE URL LIKE '%metrika%' ORDER BY EventTime LIMIT 10;;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0113
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 1
2013-09-18 00:32:23,647 Stage-1 map = 0%, reduce = 0%
2013-09-18 00:32:44,899 Stage-1 map = 1%, reduce = 0%, Cumulative CPU 144.44 sec
2013-09-18 00:32:45,910 Stage-1 map = 4%, reduce = 0%, Cumulative CPU 144.44 sec
2013-09-18 00:32:46,921 Stage-1 map = 5%, reduce = 0%, Cumulative CPU 144.44 sec
2013-09-18 00:32:47,927 Stage-1 map = 7%, reduce = 0%, Cumulative CPU 144.44 sec
2013-09-18 00:32:48,933 Stage-1 map = 9%, reduce = 0%, Cumulative CPU 144.44 sec
2013-09-18 00:32:49,944 Stage-1 map = 12%, reduce = 0%, Cumulative CPU 144.44 sec
2013-09-18 00:32:50,950 Stage-1 map = 14%, reduce = 0%, Cumulative CPU 144.44 sec
2013-09-18 00:32:51,968 Stage-1 map = 16%, reduce = 0%, Cumulative CPU 144.44 sec
2013-09-18 00:32:52,974 Stage-1 map = 18%, reduce = 0%, Cumulative CPU 144.44 sec
2013-09-18 00:32:53,981 Stage-1 map = 19%, reduce = 0%, Cumulative CPU 144.44 sec
2013-09-18 00:32:54,996 Stage-1 map = 21%, reduce = 0%, Cumulative CPU 144.44 sec
2013-09-18 00:32:56,002 Stage-1 map = 26%, reduce = 0%, Cumulative CPU 144.44 sec
2013-09-18 00:32:57,007 Stage-1 map = 28%, reduce = 0%, Cumulative CPU 144.44 sec
2013-09-18 00:32:58,206 Stage-1 map = 30%, reduce = 0%, Cumulative CPU 144.44 sec
2013-09-18 00:32:59,212 Stage-1 map = 32%, reduce = 0%, Cumulative CPU 144.44 sec
2013-09-18 00:33:00,218 Stage-1 map = 33%, reduce = 0%, Cumulative CPU 144.44 sec
2013-09-18 00:33:01,225 Stage-1 map = 34%, reduce = 0%, Cumulative CPU 144.44 sec
2013-09-18 00:33:02,242 Stage-1 map = 37%, reduce = 0%, Cumulative CPU 144.44 sec
2013-09-18 00:33:03,248 Stage-1 map = 40%, reduce = 0%, Cumulative CPU 144.44 sec
2013-09-18 00:33:04,253 Stage-1 map = 41%, reduce = 0%, Cumulative CPU 144.44 sec
2013-09-18 00:33:05,260 Stage-1 map = 44%, reduce = 0%, Cumulative CPU 144.44 sec
2013-09-18 00:33:06,266 Stage-1 map = 46%, reduce = 0%, Cumulative CPU 144.44 sec
2013-09-18 00:33:07,271 Stage-1 map = 48%, reduce = 0%, Cumulative CPU 144.44 sec
2013-09-18 00:33:08,282 Stage-1 map = 50%, reduce = 0%, Cumulative CPU 144.44 sec
2013-09-18 00:33:09,287 Stage-1 map = 53%, reduce = 0%, Cumulative CPU 144.44 sec
2013-09-18 00:33:10,300 Stage-1 map = 54%, reduce = 0%, Cumulative CPU 174.77 sec
2013-09-18 00:33:11,313 Stage-1 map = 57%, reduce = 0%, Cumulative CPU 174.77 sec
2013-09-18 00:33:12,329 Stage-1 map = 59%, reduce = 0%, Cumulative CPU 174.77 sec
2013-09-18 00:33:13,349 Stage-1 map = 59%, reduce = 0%, Cumulative CPU 174.77 sec
2013-09-18 00:33:14,361 Stage-1 map = 61%, reduce = 0%, Cumulative CPU 174.77 sec
2013-09-18 00:33:15,370 Stage-1 map = 64%, reduce = 0%, Cumulative CPU 174.77 sec
2013-09-18 00:33:16,376 Stage-1 map = 67%, reduce = 0%, Cumulative CPU 174.77 sec
2013-09-18 00:33:17,382 Stage-1 map = 69%, reduce = 0%, Cumulative CPU 174.77 sec
2013-09-18 00:33:18,389 Stage-1 map = 70%, reduce = 0%, Cumulative CPU 174.77 sec
2013-09-18 00:33:19,406 Stage-1 map = 72%, reduce = 0%, Cumulative CPU 174.77 sec
2013-09-18 00:33:20,411 Stage-1 map = 73%, reduce = 0%, Cumulative CPU 174.77 sec
2013-09-18 00:33:21,417 Stage-1 map = 75%, reduce = 0%, Cumulative CPU 174.77 sec
2013-09-18 00:33:22,426 Stage-1 map = 77%, reduce = 0%, Cumulative CPU 174.77 sec
2013-09-18 00:33:23,432 Stage-1 map = 81%, reduce = 0%, Cumulative CPU 345.6 sec
2013-09-18 00:33:24,439 Stage-1 map = 84%, reduce = 0%, Cumulative CPU 435.05 sec
2013-09-18 00:33:25,444 Stage-1 map = 86%, reduce = 0%, Cumulative CPU 477.09 sec
2013-09-18 00:33:26,450 Stage-1 map = 91%, reduce = 0%, Cumulative CPU 930.38 sec
2013-09-18 00:33:27,467 Stage-1 map = 95%, reduce = 0%, Cumulative CPU 1352.12 sec
2013-09-18 00:33:28,477 Stage-1 map = 97%, reduce = 0%, Cumulative CPU 1492.34 sec
2013-09-18 00:33:29,483 Stage-1 map = 98%, reduce = 0%, Cumulative CPU 1684.33 sec
2013-09-18 00:33:30,488 Stage-1 map = 99%, reduce = 0%, Cumulative CPU 1684.33 sec
2013-09-18 00:33:31,494 Stage-1 map = 99%, reduce = 0%, Cumulative CPU 1784.31 sec
2013-09-18 00:33:32,499 Stage-1 map = 100%, reduce = 32%, Cumulative CPU 1839.12 sec
2013-09-18 00:33:33,504 Stage-1 map = 100%, reduce = 32%, Cumulative CPU 1839.12 sec
2013-09-18 00:33:34,509 Stage-1 map = 100%, reduce = 32%, Cumulative CPU 1839.12 sec
2013-09-18 00:33:35,515 Stage-1 map = 100%, reduce = 32%, Cumulative CPU 1839.12 sec
2013-09-18 00:33:36,519 Stage-1 map = 100%, reduce = 32%, Cumulative CPU 1839.12 sec
2013-09-18 00:33:37,525 Stage-1 map = 100%, reduce = 32%, Cumulative CPU 1839.93 sec
2013-09-18 00:33:38,530 Stage-1 map = 100%, reduce = 32%, Cumulative CPU 1839.93 sec
2013-09-18 00:33:39,537 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 1842.69 sec
2013-09-18 00:33:40,543 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 1842.69 sec
MapReduce Total cumulative CPU time: 30 minutes 42 seconds 690 msec
Ended Job = job_201309172235_0113
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 1 Cumulative CPU: 1842.69 sec HDFS Read: 10081441390 HDFS Write: 5802 SUCCESS
Total MapReduce CPU Time Spent: 30 minutes 42 seconds 690 msec
OK
Time taken: 88.521 seconds, Fetched: 10 row(s)
hive> quit;
times: 3
query: SELECT * FROM hits_100m WHERE URL LIKE '%metrika%' ORDER BY EventTime LIMIT 10;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_27658@mturlrep13_201309180033_1797641182.txt
hive> SELECT * FROM hits_100m WHERE URL LIKE '%metrika%' ORDER BY EventTime LIMIT 10;;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0114
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 1
2013-09-18 00:33:53,935 Stage-1 map = 0%, reduce = 0%
2013-09-18 00:34:14,165 Stage-1 map = 1%, reduce = 0%
2013-09-18 00:34:16,361 Stage-1 map = 5%, reduce = 0%
2013-09-18 00:34:17,366 Stage-1 map = 6%, reduce = 0%
2013-09-18 00:34:18,371 Stage-1 map = 7%, reduce = 0%
2013-09-18 00:34:19,376 Stage-1 map = 10%, reduce = 0%
2013-09-18 00:34:20,389 Stage-1 map = 14%, reduce = 0%
2013-09-18 00:34:21,394 Stage-1 map = 15%, reduce = 0%
2013-09-18 00:34:22,404 Stage-1 map = 18%, reduce = 0%
2013-09-18 00:34:23,409 Stage-1 map = 19%, reduce = 0%
2013-09-18 00:34:25,642 Stage-1 map = 22%, reduce = 0%
2013-09-18 00:34:26,649 Stage-1 map = 27%, reduce = 0%
2013-09-18 00:34:28,674 Stage-1 map = 28%, reduce = 0%
2013-09-18 00:34:29,680 Stage-1 map = 31%, reduce = 0%
2013-09-18 00:34:30,686 Stage-1 map = 32%, reduce = 0%
2013-09-18 00:34:31,692 Stage-1 map = 34%, reduce = 0%
2013-09-18 00:34:32,697 Stage-1 map = 36%, reduce = 0%
2013-09-18 00:34:33,702 Stage-1 map = 39%, reduce = 0%
2013-09-18 00:34:34,719 Stage-1 map = 40%, reduce = 0%
2013-09-18 00:34:35,725 Stage-1 map = 42%, reduce = 0%
2013-09-18 00:34:36,730 Stage-1 map = 44%, reduce = 0%
2013-09-18 00:34:37,740 Stage-1 map = 45%, reduce = 0%, Cumulative CPU 1134.9 sec
2013-09-18 00:34:38,752 Stage-1 map = 48%, reduce = 0%, Cumulative CPU 1134.9 sec
2013-09-18 00:34:39,768 Stage-1 map = 49%, reduce = 0%, Cumulative CPU 1134.9 sec
2013-09-18 00:34:40,779 Stage-1 map = 50%, reduce = 0%, Cumulative CPU 1134.9 sec
2013-09-18 00:34:41,816 Stage-1 map = 54%, reduce = 0%, Cumulative CPU 1140.46 sec
2013-09-18 00:34:42,825 Stage-1 map = 57%, reduce = 0%, Cumulative CPU 1140.46 sec
2013-09-18 00:34:43,831 Stage-1 map = 59%, reduce = 0%, Cumulative CPU 1140.46 sec
2013-09-18 00:34:44,837 Stage-1 map = 61%, reduce = 0%, Cumulative CPU 1140.46 sec
2013-09-18 00:34:45,843 Stage-1 map = 63%, reduce = 0%, Cumulative CPU 1140.46 sec
2013-09-18 00:34:46,856 Stage-1 map = 63%, reduce = 0%, Cumulative CPU 1140.46 sec
2013-09-18 00:34:47,862 Stage-1 map = 66%, reduce = 0%, Cumulative CPU 1140.46 sec
2013-09-18 00:34:48,880 Stage-1 map = 70%, reduce = 0%, Cumulative CPU 1140.46 sec
2013-09-18 00:34:49,889 Stage-1 map = 71%, reduce = 0%, Cumulative CPU 1140.46 sec
2013-09-18 00:34:50,902 Stage-1 map = 73%, reduce = 0%, Cumulative CPU 1140.46 sec
2013-09-18 00:34:51,908 Stage-1 map = 76%, reduce = 0%, Cumulative CPU 1140.46 sec
2013-09-18 00:34:52,913 Stage-1 map = 77%, reduce = 0%, Cumulative CPU 1171.26 sec
2013-09-18 00:34:53,952 Stage-1 map = 80%, reduce = 0%, Cumulative CPU 1186.68 sec
2013-09-18 00:34:54,957 Stage-1 map = 83%, reduce = 0%, Cumulative CPU 1255.59 sec
2013-09-18 00:34:55,970 Stage-1 map = 87%, reduce = 0%, Cumulative CPU 1310.59 sec
2013-09-18 00:34:56,976 Stage-1 map = 90%, reduce = 0%, Cumulative CPU 1386.73 sec
2013-09-18 00:34:57,981 Stage-1 map = 93%, reduce = 0%, Cumulative CPU 1506.92 sec
2013-09-18 00:34:58,986 Stage-1 map = 97%, reduce = 0%, Cumulative CPU 1683.99 sec
2013-09-18 00:34:59,991 Stage-1 map = 98%, reduce = 0%, Cumulative CPU 1723.75 sec
2013-09-18 00:35:00,997 Stage-1 map = 98%, reduce = 0%, Cumulative CPU 1772.53 sec
2013-09-18 00:35:02,002 Stage-1 map = 99%, reduce = 0%, Cumulative CPU 1772.53 sec
2013-09-18 00:35:03,006 Stage-1 map = 99%, reduce = 0%, Cumulative CPU 1826.62 sec
2013-09-18 00:35:04,011 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1855.25 sec
2013-09-18 00:35:05,015 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1855.25 sec
2013-09-18 00:35:06,019 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1855.25 sec
2013-09-18 00:35:07,023 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1855.25 sec
2013-09-18 00:35:08,028 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1855.25 sec
2013-09-18 00:35:09,033 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 1855.25 sec
2013-09-18 00:35:10,038 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 1855.25 sec
2013-09-18 00:35:11,045 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 1858.98 sec
2013-09-18 00:35:12,051 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 1858.98 sec
MapReduce Total cumulative CPU time: 30 minutes 58 seconds 980 msec
Ended Job = job_201309172235_0114
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 1 Cumulative CPU: 1858.98 sec HDFS Read: 10081441390 HDFS Write: 5802 SUCCESS
Total MapReduce CPU Time Spent: 30 minutes 58 seconds 980 msec
OK
Time taken: 89.324 seconds, Fetched: 10 row(s)
hive> quit;
-- плохой запрос - вынимаем все столбцы.;
times: 1
query: SELECT SearchPhrase, EventTime FROM hits_100m WHERE SearchPhrase != '' ORDER BY EventTime LIMIT 10;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_637@mturlrep13_201309180035_636558816.txt
hive> SELECT SearchPhrase, EventTime FROM hits_100m WHERE SearchPhrase != '' ORDER BY EventTime LIMIT 10;;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0115
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 1
2013-09-18 00:35:29,995 Stage-1 map = 0%, reduce = 0%
2013-09-18 00:35:41,126 Stage-1 map = 1%, reduce = 0%
2013-09-18 00:35:42,132 Stage-1 map = 4%, reduce = 0%
2013-09-18 00:35:43,139 Stage-1 map = 12%, reduce = 0%
2013-09-18 00:35:44,146 Stage-1 map = 16%, reduce = 0%
2013-09-18 00:35:45,160 Stage-1 map = 18%, reduce = 0%, Cumulative CPU 8.89 sec
2013-09-18 00:35:46,172 Stage-1 map = 18%, reduce = 0%, Cumulative CPU 8.89 sec
2013-09-18 00:35:47,181 Stage-1 map = 18%, reduce = 0%, Cumulative CPU 8.89 sec
2013-09-18 00:35:48,823 Stage-1 map = 42%, reduce = 0%, Cumulative CPU 19.53 sec
2013-09-18 00:35:50,089 Stage-1 map = 52%, reduce = 0%, Cumulative CPU 92.18 sec
2013-09-18 00:35:51,103 Stage-1 map = 78%, reduce = 0%, Cumulative CPU 137.0 sec
2013-09-18 00:35:52,111 Stage-1 map = 87%, reduce = 0%, Cumulative CPU 217.4 sec
2013-09-18 00:35:53,117 Stage-1 map = 93%, reduce = 0%, Cumulative CPU 317.92 sec
2013-09-18 00:35:54,123 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 435.87 sec
2013-09-18 00:35:55,129 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 435.87 sec
2013-09-18 00:35:56,135 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 435.87 sec
2013-09-18 00:35:57,141 Stage-1 map = 100%, reduce = 6%, Cumulative CPU 435.87 sec
2013-09-18 00:35:58,148 Stage-1 map = 100%, reduce = 6%, Cumulative CPU 435.87 sec
2013-09-18 00:35:59,154 Stage-1 map = 100%, reduce = 6%, Cumulative CPU 435.87 sec
2013-09-18 00:36:00,159 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 435.87 sec
2013-09-18 00:36:01,165 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 435.87 sec
2013-09-18 00:36:02,170 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 435.87 sec
2013-09-18 00:36:03,175 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 435.87 sec
2013-09-18 00:36:04,180 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 435.87 sec
2013-09-18 00:36:05,186 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 435.87 sec
2013-09-18 00:36:06,191 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 435.87 sec
2013-09-18 00:36:07,197 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 435.87 sec
2013-09-18 00:36:08,202 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 435.87 sec
2013-09-18 00:36:09,208 Stage-1 map = 100%, reduce = 78%, Cumulative CPU 435.87 sec
2013-09-18 00:36:10,212 Stage-1 map = 100%, reduce = 78%, Cumulative CPU 435.87 sec
2013-09-18 00:36:11,217 Stage-1 map = 100%, reduce = 78%, Cumulative CPU 435.87 sec
2013-09-18 00:36:12,224 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 457.03 sec
2013-09-18 00:36:13,230 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 457.03 sec
MapReduce Total cumulative CPU time: 7 minutes 37 seconds 30 msec
Ended Job = job_201309172235_0115
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 1 Cumulative CPU: 457.03 sec HDFS Read: 184864723 HDFS Write: 747 SUCCESS
Total MapReduce CPU Time Spent: 7 minutes 37 seconds 30 msec
OK
Time taken: 53.796 seconds, Fetched: 10 row(s)
hive> quit;
times: 2
query: SELECT SearchPhrase, EventTime FROM hits_100m WHERE SearchPhrase != '' ORDER BY EventTime LIMIT 10;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_6758@mturlrep13_201309180036_334325394.txt
hive> SELECT SearchPhrase, EventTime FROM hits_100m WHERE SearchPhrase != '' ORDER BY EventTime LIMIT 10;;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0116
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 1
2013-09-18 00:36:23,502 Stage-1 map = 0%, reduce = 0%
2013-09-18 00:36:35,575 Stage-1 map = 1%, reduce = 0%
2013-09-18 00:36:36,583 Stage-1 map = 9%, reduce = 0%
2013-09-18 00:36:37,595 Stage-1 map = 16%, reduce = 0%, Cumulative CPU 232.24 sec
2013-09-18 00:36:38,607 Stage-1 map = 24%, reduce = 0%, Cumulative CPU 232.24 sec
2013-09-18 00:36:39,625 Stage-1 map = 48%, reduce = 0%, Cumulative CPU 232.24 sec
2013-09-18 00:36:40,671 Stage-1 map = 63%, reduce = 0%, Cumulative CPU 246.94 sec
2013-09-18 00:36:41,688 Stage-1 map = 78%, reduce = 0%, Cumulative CPU 294.67 sec
2013-09-18 00:36:42,699 Stage-1 map = 90%, reduce = 0%, Cumulative CPU 362.37 sec
2013-09-18 00:36:43,705 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 448.5 sec
2013-09-18 00:36:44,711 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 448.5 sec
2013-09-18 00:36:45,717 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 448.5 sec
2013-09-18 00:36:46,723 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 448.5 sec
2013-09-18 00:36:47,729 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 448.5 sec
2013-09-18 00:36:48,735 Stage-1 map = 100%, reduce = 12%, Cumulative CPU 448.5 sec
2013-09-18 00:36:49,741 Stage-1 map = 100%, reduce = 12%, Cumulative CPU 448.5 sec
2013-09-18 00:36:50,747 Stage-1 map = 100%, reduce = 12%, Cumulative CPU 448.5 sec
2013-09-18 00:36:51,753 Stage-1 map = 100%, reduce = 12%, Cumulative CPU 448.5 sec
2013-09-18 00:36:52,759 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 448.5 sec
2013-09-18 00:36:53,764 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 448.5 sec
2013-09-18 00:36:54,770 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 448.5 sec
2013-09-18 00:36:55,776 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 448.5 sec
2013-09-18 00:36:56,782 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 448.5 sec
2013-09-18 00:36:57,789 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 448.5 sec
2013-09-18 00:36:58,796 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 448.5 sec
2013-09-18 00:36:59,802 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 448.5 sec
2013-09-18 00:37:00,808 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 448.5 sec
2013-09-18 00:37:01,814 Stage-1 map = 100%, reduce = 80%, Cumulative CPU 448.5 sec
2013-09-18 00:37:02,820 Stage-1 map = 100%, reduce = 80%, Cumulative CPU 448.5 sec
2013-09-18 00:37:03,828 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 469.6 sec
2013-09-18 00:37:04,834 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 469.6 sec
2013-09-18 00:37:05,840 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 469.6 sec
MapReduce Total cumulative CPU time: 7 minutes 49 seconds 600 msec
Ended Job = job_201309172235_0116
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 1 Cumulative CPU: 469.6 sec HDFS Read: 184864723 HDFS Write: 747 SUCCESS
Total MapReduce CPU Time Spent: 7 minutes 49 seconds 600 msec
OK
Time taken: 50.592 seconds, Fetched: 10 row(s)
hive> quit;
times: 3
query: SELECT SearchPhrase, EventTime FROM hits_100m WHERE SearchPhrase != '' ORDER BY EventTime LIMIT 10;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_12112@mturlrep13_201309180037_615192245.txt
hive> SELECT SearchPhrase, EventTime FROM hits_100m WHERE SearchPhrase != '' ORDER BY EventTime LIMIT 10;;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0117
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 1
2013-09-18 00:37:16,411 Stage-1 map = 0%, reduce = 0%
2013-09-18 00:37:27,531 Stage-1 map = 1%, reduce = 0%
2013-09-18 00:37:28,541 Stage-1 map = 11%, reduce = 0%
2013-09-18 00:37:29,559 Stage-1 map = 19%, reduce = 0%
2013-09-18 00:37:30,564 Stage-1 map = 24%, reduce = 0%
2013-09-18 00:37:31,580 Stage-1 map = 51%, reduce = 0%, Cumulative CPU 9.16 sec
2013-09-18 00:37:32,592 Stage-1 map = 67%, reduce = 0%, Cumulative CPU 47.94 sec
2013-09-18 00:37:33,600 Stage-1 map = 77%, reduce = 0%, Cumulative CPU 167.83 sec
2013-09-18 00:37:34,607 Stage-1 map = 87%, reduce = 0%, Cumulative CPU 256.71 sec
2013-09-18 00:37:35,612 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 445.74 sec
2013-09-18 00:37:36,617 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 445.74 sec
2013-09-18 00:37:37,622 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 445.74 sec
2013-09-18 00:37:38,627 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 445.74 sec
2013-09-18 00:37:39,632 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 445.74 sec
2013-09-18 00:37:40,637 Stage-1 map = 100%, reduce = 11%, Cumulative CPU 445.74 sec
2013-09-18 00:37:41,643 Stage-1 map = 100%, reduce = 11%, Cumulative CPU 445.74 sec
2013-09-18 00:37:42,648 Stage-1 map = 100%, reduce = 11%, Cumulative CPU 445.74 sec
2013-09-18 00:37:43,654 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 445.74 sec
2013-09-18 00:37:44,659 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 445.74 sec
2013-09-18 00:37:45,664 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 445.74 sec
2013-09-18 00:37:46,669 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 445.74 sec
2013-09-18 00:37:47,675 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 445.74 sec
2013-09-18 00:37:48,680 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 445.74 sec
2013-09-18 00:37:49,685 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 445.74 sec
2013-09-18 00:37:50,691 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 445.74 sec
2013-09-18 00:37:51,696 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 445.74 sec
2013-09-18 00:37:52,701 Stage-1 map = 100%, reduce = 78%, Cumulative CPU 445.74 sec
2013-09-18 00:37:53,706 Stage-1 map = 100%, reduce = 78%, Cumulative CPU 445.74 sec
2013-09-18 00:37:54,711 Stage-1 map = 100%, reduce = 78%, Cumulative CPU 445.74 sec
2013-09-18 00:37:55,718 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 467.25 sec
2013-09-18 00:37:56,723 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 467.25 sec
2013-09-18 00:37:57,729 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 467.25 sec
MapReduce Total cumulative CPU time: 7 minutes 47 seconds 250 msec
Ended Job = job_201309172235_0117
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 1 Cumulative CPU: 467.25 sec HDFS Read: 184864723 HDFS Write: 747 SUCCESS
Total MapReduce CPU Time Spent: 7 minutes 47 seconds 250 msec
OK
Time taken: 50.118 seconds, Fetched: 10 row(s)
hive> quit;
-- большая сортировка.;
times: 1
query: SELECT SearchPhrase FROM hits_100m WHERE SearchPhrase != '' ORDER BY SearchPhrase LIMIT 10;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_17453@mturlrep13_201309180038_1517091963.txt
hive> SELECT SearchPhrase FROM hits_100m WHERE SearchPhrase != '' ORDER BY SearchPhrase LIMIT 10;;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0118
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 1
2013-09-18 00:38:15,283 Stage-1 map = 0%, reduce = 0%
2013-09-18 00:38:25,343 Stage-1 map = 1%, reduce = 0%
2013-09-18 00:38:27,409 Stage-1 map = 4%, reduce = 0%
2013-09-18 00:38:28,422 Stage-1 map = 14%, reduce = 0%
2013-09-18 00:38:29,428 Stage-1 map = 20%, reduce = 0%
2013-09-18 00:38:31,442 Stage-1 map = 46%, reduce = 0%
2013-09-18 00:38:32,447 Stage-1 map = 54%, reduce = 0%
2013-09-18 00:38:34,502 Stage-1 map = 75%, reduce = 0%, Cumulative CPU 56.43 sec
2013-09-18 00:38:35,526 Stage-1 map = 82%, reduce = 0%, Cumulative CPU 160.97 sec
2013-09-18 00:38:36,540 Stage-1 map = 86%, reduce = 0%, Cumulative CPU 264.56 sec
2013-09-18 00:38:37,546 Stage-1 map = 96%, reduce = 0%, Cumulative CPU 421.84 sec
2013-09-18 00:38:38,552 Stage-1 map = 98%, reduce = 0%, Cumulative CPU 433.68 sec
2013-09-18 00:38:39,557 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 442.48 sec
2013-09-18 00:38:40,562 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 442.48 sec
2013-09-18 00:38:41,567 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 442.48 sec
2013-09-18 00:38:42,573 Stage-1 map = 100%, reduce = 5%, Cumulative CPU 442.48 sec
2013-09-18 00:38:43,579 Stage-1 map = 100%, reduce = 5%, Cumulative CPU 442.48 sec
2013-09-18 00:38:44,585 Stage-1 map = 100%, reduce = 5%, Cumulative CPU 442.48 sec
2013-09-18 00:38:45,590 Stage-1 map = 100%, reduce = 30%, Cumulative CPU 442.48 sec
2013-09-18 00:38:46,596 Stage-1 map = 100%, reduce = 30%, Cumulative CPU 442.48 sec
2013-09-18 00:38:47,602 Stage-1 map = 100%, reduce = 30%, Cumulative CPU 442.48 sec
2013-09-18 00:38:48,607 Stage-1 map = 100%, reduce = 30%, Cumulative CPU 442.48 sec
2013-09-18 00:38:49,613 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 442.48 sec
2013-09-18 00:38:50,618 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 442.48 sec
2013-09-18 00:38:51,624 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 442.48 sec
2013-09-18 00:38:52,629 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 442.48 sec
2013-09-18 00:38:53,635 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 442.48 sec
2013-09-18 00:38:54,640 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 442.48 sec
2013-09-18 00:38:55,645 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 442.48 sec
2013-09-18 00:38:56,649 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 442.48 sec
2013-09-18 00:38:57,654 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 442.48 sec
2013-09-18 00:38:58,658 Stage-1 map = 100%, reduce = 67%, Cumulative CPU 442.48 sec
2013-09-18 00:38:59,663 Stage-1 map = 100%, reduce = 67%, Cumulative CPU 442.48 sec
2013-09-18 00:39:00,668 Stage-1 map = 100%, reduce = 67%, Cumulative CPU 442.48 sec
2013-09-18 00:39:01,673 Stage-1 map = 100%, reduce = 82%, Cumulative CPU 442.48 sec
2013-09-18 00:39:02,677 Stage-1 map = 100%, reduce = 82%, Cumulative CPU 442.48 sec
2013-09-18 00:39:03,682 Stage-1 map = 100%, reduce = 82%, Cumulative CPU 442.48 sec
2013-09-18 00:39:04,689 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 477.8 sec
2013-09-18 00:39:08,034 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 477.8 sec
2013-09-18 00:39:09,045 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 477.8 sec
2013-09-18 00:39:10,050 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 477.8 sec
2013-09-18 00:39:11,056 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 477.8 sec
2013-09-18 00:39:12,062 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 477.8 sec
MapReduce Total cumulative CPU time: 7 minutes 57 seconds 800 msec
Ended Job = job_201309172235_0118
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 1 Cumulative CPU: 477.8 sec HDFS Read: 182281719 HDFS Write: 509 SUCCESS
Total MapReduce CPU Time Spent: 7 minutes 57 seconds 800 msec
OK
 снип 2.07.01-89
 19105-79
! 20 самых ужасные пыток в истории человечества. видео
! bt
! гобилен желтый купить )
! жизнь в тюрьме россии -
! минимальное значение измеряемой величины?
! новый htc 603e
! о чем говорит огромное количество антител в крови к кишечной палочке
Time taken: 68.786 seconds, Fetched: 10 row(s)
hive> quit;
times: 2
query: SELECT SearchPhrase FROM hits_100m WHERE SearchPhrase != '' ORDER BY SearchPhrase LIMIT 10;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_22553@mturlrep13_201309180039_34144715.txt
hive> SELECT SearchPhrase FROM hits_100m WHERE SearchPhrase != '' ORDER BY SearchPhrase LIMIT 10;;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0119
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 1
2013-09-18 00:39:23,346 Stage-1 map = 0%, reduce = 0%
2013-09-18 00:39:35,450 Stage-1 map = 2%, reduce = 0%
2013-09-18 00:39:36,456 Stage-1 map = 19%, reduce = 0%
2013-09-18 00:39:37,472 Stage-1 map = 33%, reduce = 0%, Cumulative CPU 259.71 sec
2013-09-18 00:39:38,480 Stage-1 map = 34%, reduce = 0%, Cumulative CPU 259.71 sec
2013-09-18 00:39:39,499 Stage-1 map = 55%, reduce = 0%, Cumulative CPU 267.32 sec
2013-09-18 00:39:40,516 Stage-1 map = 74%, reduce = 0%, Cumulative CPU 282.8 sec
2013-09-18 00:39:41,528 Stage-1 map = 81%, reduce = 0%, Cumulative CPU 308.59 sec
2013-09-18 00:39:42,536 Stage-1 map = 95%, reduce = 0%, Cumulative CPU 377.66 sec
2013-09-18 00:39:43,542 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 452.74 sec
2013-09-18 00:39:44,548 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 452.74 sec
2013-09-18 00:39:45,553 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 452.74 sec
2013-09-18 00:39:46,559 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 452.74 sec
2013-09-18 00:39:47,565 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 452.74 sec
2013-09-18 00:39:48,572 Stage-1 map = 100%, reduce = 9%, Cumulative CPU 452.74 sec
2013-09-18 00:39:49,578 Stage-1 map = 100%, reduce = 9%, Cumulative CPU 452.74 sec
2013-09-18 00:39:50,584 Stage-1 map = 100%, reduce = 9%, Cumulative CPU 452.74 sec
2013-09-18 00:39:51,590 Stage-1 map = 100%, reduce = 29%, Cumulative CPU 452.74 sec
2013-09-18 00:39:52,596 Stage-1 map = 100%, reduce = 29%, Cumulative CPU 452.74 sec
2013-09-18 00:39:53,602 Stage-1 map = 100%, reduce = 29%, Cumulative CPU 452.74 sec
2013-09-18 00:39:54,607 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 452.74 sec
2013-09-18 00:39:55,613 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 452.74 sec
2013-09-18 00:39:56,619 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 452.74 sec
2013-09-18 00:39:57,625 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 452.74 sec
2013-09-18 00:39:58,630 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 452.74 sec
2013-09-18 00:39:59,636 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 452.74 sec
2013-09-18 00:40:00,642 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 452.74 sec
2013-09-18 00:40:01,648 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 452.74 sec
2013-09-18 00:40:02,655 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 452.74 sec
2013-09-18 00:40:03,660 Stage-1 map = 100%, reduce = 67%, Cumulative CPU 452.74 sec
2013-09-18 00:40:04,665 Stage-1 map = 100%, reduce = 67%, Cumulative CPU 452.74 sec
2013-09-18 00:40:05,670 Stage-1 map = 100%, reduce = 67%, Cumulative CPU 452.74 sec
2013-09-18 00:40:06,675 Stage-1 map = 100%, reduce = 80%, Cumulative CPU 452.74 sec
2013-09-18 00:40:07,680 Stage-1 map = 100%, reduce = 80%, Cumulative CPU 452.74 sec
2013-09-18 00:40:08,686 Stage-1 map = 100%, reduce = 80%, Cumulative CPU 452.74 sec
2013-09-18 00:40:09,691 Stage-1 map = 100%, reduce = 96%, Cumulative CPU 452.74 sec
2013-09-18 00:40:10,699 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 487.91 sec
2013-09-18 00:40:11,704 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 487.91 sec
2013-09-18 00:40:12,710 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 487.91 sec
MapReduce Total cumulative CPU time: 8 minutes 7 seconds 910 msec
Ended Job = job_201309172235_0119
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 1 Cumulative CPU: 487.91 sec HDFS Read: 182281719 HDFS Write: 509 SUCCESS
Total MapReduce CPU Time Spent: 8 minutes 7 seconds 910 msec
OK
 снип 2.07.01-89
 19105-79
! 20 самых ужасные пыток в истории человечества. видео
! bt
! гобилен желтый купить )
! жизнь в тюрьме россии -
! минимальное значение измеряемой величины?
! новый htc 603e
! о чем говорит огромное количество антител в крови к кишечной палочке
Time taken: 63.993 seconds, Fetched: 10 row(s)
hive> quit;
times: 3
query: SELECT SearchPhrase FROM hits_100m WHERE SearchPhrase != '' ORDER BY SearchPhrase LIMIT 10;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_27974@mturlrep13_201309180040_814853077.txt
hive> SELECT SearchPhrase FROM hits_100m WHERE SearchPhrase != '' ORDER BY SearchPhrase LIMIT 10;;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0120
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 1
2013-09-18 00:40:30,016 Stage-1 map = 0%, reduce = 0%
2013-09-18 00:40:42,091 Stage-1 map = 12%, reduce = 0%
2013-09-18 00:40:43,103 Stage-1 map = 28%, reduce = 0%
2013-09-18 00:40:44,113 Stage-1 map = 34%, reduce = 0%
2013-09-18 00:40:45,135 Stage-1 map = 52%, reduce = 0%, Cumulative CPU 18.32 sec
2013-09-18 00:40:46,146 Stage-1 map = 73%, reduce = 0%, Cumulative CPU 77.03 sec
2013-09-18 00:40:47,154 Stage-1 map = 80%, reduce = 0%, Cumulative CPU 151.48 sec
2013-09-18 00:40:48,161 Stage-1 map = 85%, reduce = 0%, Cumulative CPU 201.55 sec
2013-09-18 00:40:49,167 Stage-1 map = 98%, reduce = 0%, Cumulative CPU 395.4 sec
2013-09-18 00:40:50,173 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 452.5 sec
2013-09-18 00:40:51,178 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 452.5 sec
2013-09-18 00:40:52,184 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 452.5 sec
2013-09-18 00:40:53,259 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 452.5 sec
2013-09-18 00:40:54,265 Stage-1 map = 100%, reduce = 7%, Cumulative CPU 452.5 sec
2013-09-18 00:40:55,271 Stage-1 map = 100%, reduce = 7%, Cumulative CPU 452.5 sec
2013-09-18 00:40:56,281 Stage-1 map = 100%, reduce = 7%, Cumulative CPU 452.5 sec
2013-09-18 00:40:57,288 Stage-1 map = 100%, reduce = 30%, Cumulative CPU 452.5 sec
2013-09-18 00:40:58,293 Stage-1 map = 100%, reduce = 30%, Cumulative CPU 452.5 sec
2013-09-18 00:40:59,298 Stage-1 map = 100%, reduce = 30%, Cumulative CPU 452.5 sec
2013-09-18 00:41:00,303 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 452.5 sec
2013-09-18 00:41:01,307 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 452.5 sec
2013-09-18 00:41:02,312 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 452.5 sec
2013-09-18 00:41:03,317 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 452.5 sec
2013-09-18 00:41:04,322 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 452.5 sec
2013-09-18 00:41:05,326 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 452.5 sec
2013-09-18 00:41:06,331 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 452.5 sec
2013-09-18 00:41:07,336 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 452.5 sec
2013-09-18 00:41:08,341 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 452.5 sec
2013-09-18 00:41:09,346 Stage-1 map = 100%, reduce = 67%, Cumulative CPU 452.5 sec
2013-09-18 00:41:10,351 Stage-1 map = 100%, reduce = 67%, Cumulative CPU 452.5 sec
2013-09-18 00:41:11,356 Stage-1 map = 100%, reduce = 67%, Cumulative CPU 452.5 sec
2013-09-18 00:41:12,360 Stage-1 map = 100%, reduce = 80%, Cumulative CPU 452.5 sec
2013-09-18 00:41:13,365 Stage-1 map = 100%, reduce = 80%, Cumulative CPU 452.5 sec
2013-09-18 00:41:14,369 Stage-1 map = 100%, reduce = 80%, Cumulative CPU 452.5 sec
2013-09-18 00:41:15,374 Stage-1 map = 100%, reduce = 97%, Cumulative CPU 452.5 sec
2013-09-18 00:41:16,381 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 487.48 sec
2013-09-18 00:41:17,387 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 487.48 sec
2013-09-18 00:41:18,392 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 487.48 sec
MapReduce Total cumulative CPU time: 8 minutes 7 seconds 480 msec
Ended Job = job_201309172235_0120
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 1 Cumulative CPU: 487.48 sec HDFS Read: 182281719 HDFS Write: 509 SUCCESS
Total MapReduce CPU Time Spent: 8 minutes 7 seconds 480 msec
OK
 снип 2.07.01-89
 19105-79
! 20 самых ужасные пыток в истории человечества. видео
! bt
! гобилен желтый купить )
! жизнь в тюрьме россии -
! минимальное значение измеряемой величины?
! новый htc 603e
! о чем говорит огромное количество антител в крови к кишечной палочке
Time taken: 62.886 seconds, Fetched: 10 row(s)
hive> quit;
-- большая сортировка по строкам.;
times: 1
query: SELECT SearchPhrase, EventTime FROM hits_100m WHERE SearchPhrase != '' ORDER BY EventTime, SearchPhrase LIMIT 10;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_964@mturlrep13_201309180041_1295667471.txt
hive> SELECT SearchPhrase, EventTime FROM hits_100m WHERE SearchPhrase != '' ORDER BY EventTime, SearchPhrase LIMIT 10;;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0121
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 1
2013-09-18 00:41:41,880 Stage-1 map = 0%, reduce = 0%
2013-09-18 00:41:53,981 Stage-1 map = 3%, reduce = 0%
2013-09-18 00:41:54,989 Stage-1 map = 10%, reduce = 0%
2013-09-18 00:41:55,998 Stage-1 map = 16%, reduce = 0%
2013-09-18 00:41:57,005 Stage-1 map = 24%, reduce = 0%
2013-09-18 00:41:58,010 Stage-1 map = 34%, reduce = 0%
2013-09-18 00:41:59,015 Stage-1 map = 39%, reduce = 0%
2013-09-18 00:42:00,845 Stage-1 map = 51%, reduce = 0%
2013-09-18 00:42:02,853 Stage-1 map = 68%, reduce = 0%, Cumulative CPU 116.0 sec
2013-09-18 00:42:03,921 Stage-1 map = 85%, reduce = 0%, Cumulative CPU 228.22 sec
2013-09-18 00:42:04,929 Stage-1 map = 92%, reduce = 0%, Cumulative CPU 296.35 sec
2013-09-18 00:42:05,935 Stage-1 map = 97%, reduce = 0%, Cumulative CPU 412.74 sec
2013-09-18 00:42:06,941 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 470.11 sec
2013-09-18 00:42:07,947 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 470.11 sec
2013-09-18 00:42:08,953 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 470.11 sec
2013-09-18 00:42:09,962 Stage-1 map = 100%, reduce = 5%, Cumulative CPU 470.11 sec
2013-09-18 00:42:10,968 Stage-1 map = 100%, reduce = 5%, Cumulative CPU 470.11 sec
2013-09-18 00:42:11,974 Stage-1 map = 100%, reduce = 5%, Cumulative CPU 470.11 sec
2013-09-18 00:42:12,980 Stage-1 map = 100%, reduce = 27%, Cumulative CPU 470.11 sec
2013-09-18 00:42:13,985 Stage-1 map = 100%, reduce = 27%, Cumulative CPU 470.11 sec
2013-09-18 00:42:14,991 Stage-1 map = 100%, reduce = 27%, Cumulative CPU 470.11 sec
2013-09-18 00:42:15,996 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 470.11 sec
2013-09-18 00:42:17,001 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 470.11 sec
2013-09-18 00:42:18,007 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 470.11 sec
2013-09-18 00:42:19,012 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 470.11 sec
2013-09-18 00:42:20,018 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 470.11 sec
2013-09-18 00:42:21,024 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 470.11 sec
2013-09-18 00:42:22,030 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 470.11 sec
2013-09-18 00:42:23,036 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 470.11 sec
2013-09-18 00:42:24,040 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 470.11 sec
2013-09-18 00:42:25,044 Stage-1 map = 100%, reduce = 74%, Cumulative CPU 470.11 sec
2013-09-18 00:42:26,049 Stage-1 map = 100%, reduce = 74%, Cumulative CPU 470.11 sec
2013-09-18 00:42:27,054 Stage-1 map = 100%, reduce = 74%, Cumulative CPU 470.11 sec
2013-09-18 00:42:28,059 Stage-1 map = 100%, reduce = 90%, Cumulative CPU 470.11 sec
2013-09-18 00:42:29,064 Stage-1 map = 100%, reduce = 90%, Cumulative CPU 470.11 sec
2013-09-18 00:42:30,071 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 503.72 sec
2013-09-18 00:42:31,077 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 503.72 sec
2013-09-18 00:42:32,083 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 503.72 sec
MapReduce Total cumulative CPU time: 8 minutes 23 seconds 720 msec
Ended Job = job_201309172235_0121
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 1 Cumulative CPU: 503.72 sec HDFS Read: 184864723 HDFS Write: 758 SUCCESS
Total MapReduce CPU Time Spent: 8 minutes 23 seconds 720 msec
OK
Time taken: 60.715 seconds, Fetched: 10 row(s)
hive> quit;
times: 2
query: SELECT SearchPhrase, EventTime FROM hits_100m WHERE SearchPhrase != '' ORDER BY EventTime, SearchPhrase LIMIT 10;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_6757@mturlrep13_201309180042_1836094631.txt
hive> SELECT SearchPhrase, EventTime FROM hits_100m WHERE SearchPhrase != '' ORDER BY EventTime, SearchPhrase LIMIT 10;;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0122
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 1
2013-09-18 00:42:42,848 Stage-1 map = 0%, reduce = 0%
2013-09-18 00:42:53,935 Stage-1 map = 1%, reduce = 0%
2013-09-18 00:42:54,967 Stage-1 map = 7%, reduce = 0%
2013-09-18 00:42:55,986 Stage-1 map = 15%, reduce = 0%
2013-09-18 00:42:56,991 Stage-1 map = 19%, reduce = 0%
2013-09-18 00:42:58,003 Stage-1 map = 37%, reduce = 0%
2013-09-18 00:42:59,028 Stage-1 map = 58%, reduce = 0%, Cumulative CPU 20.21 sec
2013-09-18 00:43:00,042 Stage-1 map = 71%, reduce = 0%, Cumulative CPU 95.48 sec
2013-09-18 00:43:01,060 Stage-1 map = 88%, reduce = 0%, Cumulative CPU 203.64 sec
2013-09-18 00:43:02,067 Stage-1 map = 94%, reduce = 0%, Cumulative CPU 316.18 sec
2013-09-18 00:43:03,074 Stage-1 map = 99%, reduce = 0%, Cumulative CPU 451.06 sec
2013-09-18 00:43:04,079 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 482.61 sec
2013-09-18 00:43:05,085 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 482.61 sec
2013-09-18 00:43:06,090 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 482.61 sec
2013-09-18 00:43:07,095 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 482.61 sec
2013-09-18 00:43:08,100 Stage-1 map = 100%, reduce = 6%, Cumulative CPU 482.61 sec
2013-09-18 00:43:09,105 Stage-1 map = 100%, reduce = 6%, Cumulative CPU 482.61 sec
2013-09-18 00:43:10,112 Stage-1 map = 100%, reduce = 6%, Cumulative CPU 482.61 sec
2013-09-18 00:43:11,119 Stage-1 map = 100%, reduce = 28%, Cumulative CPU 482.61 sec
2013-09-18 00:43:12,124 Stage-1 map = 100%, reduce = 28%, Cumulative CPU 482.61 sec
2013-09-18 00:43:13,130 Stage-1 map = 100%, reduce = 28%, Cumulative CPU 482.61 sec
2013-09-18 00:43:14,136 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 482.61 sec
2013-09-18 00:43:15,141 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 482.61 sec
2013-09-18 00:43:16,147 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 482.61 sec
2013-09-18 00:43:17,152 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 482.61 sec
2013-09-18 00:43:18,158 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 482.61 sec
2013-09-18 00:43:19,162 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 482.61 sec
2013-09-18 00:43:20,167 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 482.61 sec
2013-09-18 00:43:21,172 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 482.61 sec
2013-09-18 00:43:22,177 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 482.61 sec
2013-09-18 00:43:23,182 Stage-1 map = 100%, reduce = 77%, Cumulative CPU 482.61 sec
2013-09-18 00:43:24,186 Stage-1 map = 100%, reduce = 77%, Cumulative CPU 482.61 sec
2013-09-18 00:43:25,190 Stage-1 map = 100%, reduce = 77%, Cumulative CPU 482.61 sec
2013-09-18 00:43:26,195 Stage-1 map = 100%, reduce = 93%, Cumulative CPU 482.61 sec
2013-09-18 00:43:27,200 Stage-1 map = 100%, reduce = 93%, Cumulative CPU 482.61 sec
2013-09-18 00:43:28,206 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 517.47 sec
2013-09-18 00:43:29,827 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 517.47 sec
MapReduce Total cumulative CPU time: 8 minutes 37 seconds 470 msec
Ended Job = job_201309172235_0122
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 1 Cumulative CPU: 517.47 sec HDFS Read: 184864723 HDFS Write: 758 SUCCESS
Total MapReduce CPU Time Spent: 8 minutes 37 seconds 470 msec
OK
Time taken: 56.14 seconds, Fetched: 10 row(s)
hive> quit;
times: 3
query: SELECT SearchPhrase, EventTime FROM hits_100m WHERE SearchPhrase != '' ORDER BY EventTime, SearchPhrase LIMIT 10;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_12116@mturlrep13_201309180043_688174400.txt
hive> SELECT SearchPhrase, EventTime FROM hits_100m WHERE SearchPhrase != '' ORDER BY EventTime, SearchPhrase LIMIT 10;;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0123
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 1
2013-09-18 00:43:39,885 Stage-1 map = 0%, reduce = 0%
2013-09-18 00:43:52,084 Stage-1 map = 1%, reduce = 0%
2013-09-18 00:43:53,089 Stage-1 map = 9%, reduce = 0%
2013-09-18 00:43:54,103 Stage-1 map = 21%, reduce = 0%
2013-09-18 00:43:55,109 Stage-1 map = 22%, reduce = 0%
2013-09-18 00:43:56,124 Stage-1 map = 47%, reduce = 0%, Cumulative CPU 9.78 sec
2013-09-18 00:43:57,139 Stage-1 map = 63%, reduce = 0%, Cumulative CPU 49.2 sec
2013-09-18 00:43:58,156 Stage-1 map = 73%, reduce = 0%, Cumulative CPU 146.89 sec
2013-09-18 00:43:59,166 Stage-1 map = 89%, reduce = 0%, Cumulative CPU 211.3 sec
2013-09-18 00:44:00,173 Stage-1 map = 96%, reduce = 0%, Cumulative CPU 339.17 sec
2013-09-18 00:44:01,179 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 474.94 sec
2013-09-18 00:44:02,184 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 474.94 sec
2013-09-18 00:44:03,190 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 474.94 sec
2013-09-18 00:44:04,196 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 474.94 sec
2013-09-18 00:44:05,203 Stage-1 map = 100%, reduce = 8%, Cumulative CPU 474.94 sec
2013-09-18 00:44:06,209 Stage-1 map = 100%, reduce = 8%, Cumulative CPU 474.94 sec
2013-09-18 00:44:07,216 Stage-1 map = 100%, reduce = 8%, Cumulative CPU 474.94 sec
2013-09-18 00:44:08,222 Stage-1 map = 100%, reduce = 28%, Cumulative CPU 474.94 sec
2013-09-18 00:44:09,228 Stage-1 map = 100%, reduce = 28%, Cumulative CPU 474.94 sec
2013-09-18 00:44:10,235 Stage-1 map = 100%, reduce = 28%, Cumulative CPU 474.94 sec
2013-09-18 00:44:11,241 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 474.94 sec
2013-09-18 00:44:12,247 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 474.94 sec
2013-09-18 00:44:13,253 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 474.94 sec
2013-09-18 00:44:14,258 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 474.94 sec
2013-09-18 00:44:15,264 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 474.94 sec
2013-09-18 00:44:16,270 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 474.94 sec
2013-09-18 00:44:17,277 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 474.94 sec
2013-09-18 00:44:18,283 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 474.94 sec
2013-09-18 00:44:19,290 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 474.94 sec
2013-09-18 00:44:20,296 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 474.94 sec
2013-09-18 00:44:21,301 Stage-1 map = 100%, reduce = 75%, Cumulative CPU 474.94 sec
2013-09-18 00:44:22,306 Stage-1 map = 100%, reduce = 75%, Cumulative CPU 474.94 sec
2013-09-18 00:44:23,312 Stage-1 map = 100%, reduce = 75%, Cumulative CPU 474.94 sec
2013-09-18 00:44:24,318 Stage-1 map = 100%, reduce = 92%, Cumulative CPU 474.94 sec
2013-09-18 00:44:25,324 Stage-1 map = 100%, reduce = 92%, Cumulative CPU 474.94 sec
2013-09-18 00:44:26,332 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 511.22 sec
2013-09-18 00:44:27,337 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 511.22 sec
MapReduce Total cumulative CPU time: 8 minutes 31 seconds 220 msec
Ended Job = job_201309172235_0123
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 1 Cumulative CPU: 511.22 sec HDFS Read: 184864723 HDFS Write: 758 SUCCESS
Total MapReduce CPU Time Spent: 8 minutes 31 seconds 220 msec
OK
Time taken: 55.303 seconds, Fetched: 10 row(s)
hive> quit;
-- большая сортировка по кортежу.;
times: 1
query: SELECT CounterID, avg(length(URL)) AS l, count(*) AS c FROM hits_100m WHERE URL != '' GROUP BY CounterID HAVING count(*) > 100000 ORDER BY l DESC LIMIT 25;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_17460@mturlrep13_201309180044_876855052.txt
hive> SELECT CounterID, avg(length(URL)) AS l, count(*) AS c FROM hits_100m WHERE URL != '' GROUP BY CounterID HAVING count(*) > 100000 ORDER BY l DESC LIMIT 25;;
Total MapReduce jobs = 2
Launching Job 1 out of 2
Number of reduce tasks not specified. Estimated from input data size: 11
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0124
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 11
2013-09-18 00:44:53,239 Stage-1 map = 0%, reduce = 0%
2013-09-18 00:45:09,346 Stage-1 map = 1%, reduce = 0%
2013-09-18 00:45:10,353 Stage-1 map = 7%, reduce = 0%
2013-09-18 00:45:11,358 Stage-1 map = 8%, reduce = 0%
2013-09-18 00:45:12,365 Stage-1 map = 11%, reduce = 0%
2013-09-18 00:45:14,374 Stage-1 map = 14%, reduce = 0%
2013-09-18 00:45:15,389 Stage-1 map = 17%, reduce = 0%
2013-09-18 00:45:16,396 Stage-1 map = 21%, reduce = 0%
2013-09-18 00:45:17,410 Stage-1 map = 25%, reduce = 0%
2013-09-18 00:45:18,415 Stage-1 map = 28%, reduce = 0%
2013-09-18 00:45:19,438 Stage-1 map = 33%, reduce = 0%, Cumulative CPU 14.93 sec
2013-09-18 00:45:20,445 Stage-1 map = 36%, reduce = 0%, Cumulative CPU 14.93 sec
2013-09-18 00:45:22,432 Stage-1 map = 40%, reduce = 0%, Cumulative CPU 14.93 sec
2013-09-18 00:45:23,470 Stage-1 map = 45%, reduce = 0%, Cumulative CPU 14.93 sec
2013-09-18 00:45:24,485 Stage-1 map = 45%, reduce = 0%, Cumulative CPU 14.93 sec
2013-09-18 00:45:25,493 Stage-1 map = 53%, reduce = 0%, Cumulative CPU 14.93 sec
2013-09-18 00:45:27,100 Stage-1 map = 57%, reduce = 0%, Cumulative CPU 14.93 sec
2013-09-18 00:45:28,112 Stage-1 map = 63%, reduce = 0%, Cumulative CPU 14.93 sec
2013-09-18 00:45:29,121 Stage-1 map = 67%, reduce = 0%, Cumulative CPU 14.93 sec
2013-09-18 00:45:30,130 Stage-1 map = 72%, reduce = 0%, Cumulative CPU 14.93 sec
2013-09-18 00:45:31,442 Stage-1 map = 78%, reduce = 0%, Cumulative CPU 14.93 sec
2013-09-18 00:45:32,466 Stage-1 map = 81%, reduce = 0%, Cumulative CPU 14.93 sec
2013-09-18 00:45:33,482 Stage-1 map = 85%, reduce = 0%, Cumulative CPU 72.87 sec
2013-09-18 00:45:34,504 Stage-1 map = 88%, reduce = 0%, Cumulative CPU 152.52 sec
2013-09-18 00:45:35,718 Stage-1 map = 91%, reduce = 0%, Cumulative CPU 324.07 sec
2013-09-18 00:45:36,726 Stage-1 map = 94%, reduce = 0%, Cumulative CPU 615.07 sec
2013-09-18 00:45:37,738 Stage-1 map = 97%, reduce = 0%, Cumulative CPU 861.87 sec
2013-09-18 00:45:38,744 Stage-1 map = 99%, reduce = 0%, Cumulative CPU 1085.12 sec
2013-09-18 00:45:39,750 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1087.12 sec
2013-09-18 00:45:40,756 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1087.12 sec
2013-09-18 00:45:41,761 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1087.12 sec
2013-09-18 00:45:42,772 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1087.12 sec
2013-09-18 00:45:43,781 Stage-1 map = 100%, reduce = 8%, Cumulative CPU 1087.12 sec
2013-09-18 00:45:44,801 Stage-1 map = 100%, reduce = 12%, Cumulative CPU 1087.12 sec
2013-09-18 00:45:45,818 Stage-1 map = 100%, reduce = 15%, Cumulative CPU 1087.12 sec
2013-09-18 00:45:46,825 Stage-1 map = 100%, reduce = 28%, Cumulative CPU 1087.12 sec
2013-09-18 00:45:47,831 Stage-1 map = 100%, reduce = 36%, Cumulative CPU 1087.12 sec
2013-09-18 00:45:48,838 Stage-1 map = 100%, reduce = 39%, Cumulative CPU 1087.12 sec
2013-09-18 00:45:49,845 Stage-1 map = 100%, reduce = 44%, Cumulative CPU 1087.12 sec
2013-09-18 00:45:50,852 Stage-1 map = 100%, reduce = 47%, Cumulative CPU 1087.12 sec
2013-09-18 00:45:51,861 Stage-1 map = 100%, reduce = 49%, Cumulative CPU 1098.67 sec
2013-09-18 00:45:52,867 Stage-1 map = 100%, reduce = 60%, Cumulative CPU 1098.67 sec
2013-09-18 00:45:53,875 Stage-1 map = 100%, reduce = 74%, Cumulative CPU 1098.67 sec
2013-09-18 00:45:54,882 Stage-1 map = 100%, reduce = 74%, Cumulative CPU 1098.67 sec
2013-09-18 00:45:55,889 Stage-1 map = 100%, reduce = 78%, Cumulative CPU 1114.66 sec
2013-09-18 00:45:56,895 Stage-1 map = 100%, reduce = 82%, Cumulative CPU 1114.66 sec
2013-09-18 00:45:57,902 Stage-1 map = 100%, reduce = 83%, Cumulative CPU 1132.03 sec
2013-09-18 00:45:58,908 Stage-1 map = 100%, reduce = 85%, Cumulative CPU 1132.03 sec
2013-09-18 00:45:59,915 Stage-1 map = 100%, reduce = 89%, Cumulative CPU 1132.03 sec
2013-09-18 00:46:00,922 Stage-1 map = 100%, reduce = 90%, Cumulative CPU 1175.97 sec
2013-09-18 00:46:01,929 Stage-1 map = 100%, reduce = 92%, Cumulative CPU 1199.97 sec
2013-09-18 00:46:02,936 Stage-1 map = 100%, reduce = 94%, Cumulative CPU 1199.97 sec
2013-09-18 00:46:03,943 Stage-1 map = 100%, reduce = 95%, Cumulative CPU 1199.97 sec
2013-09-18 00:46:04,949 Stage-1 map = 100%, reduce = 95%, Cumulative CPU 1199.97 sec
2013-09-18 00:46:05,955 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1199.97 sec
2013-09-18 00:46:10,612 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1224.95 sec
2013-09-18 00:46:11,618 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1224.95 sec
2013-09-18 00:46:13,271 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1250.63 sec
2013-09-18 00:46:14,276 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1250.63 sec
2013-09-18 00:46:15,344 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1277.48 sec
2013-09-18 00:46:16,350 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 1333.41 sec
2013-09-18 00:46:17,356 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 1333.41 sec
2013-09-18 00:46:20,396 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 1333.41 sec
MapReduce Total cumulative CPU time: 22 minutes 13 seconds 410 msec
Ended Job = job_201309172235_0124
Launching Job 2 out of 2
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0125
Hadoop job information for Stage-2: number of mappers: 1; number of reducers: 1
2013-09-18 00:46:24,314 Stage-2 map = 0%, reduce = 0%
2013-09-18 00:46:25,319 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.71 sec
2013-09-18 00:46:26,325 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.71 sec
2013-09-18 00:46:27,330 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.71 sec
2013-09-18 00:46:28,335 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.71 sec
2013-09-18 00:46:29,340 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.71 sec
2013-09-18 00:46:30,345 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.71 sec
2013-09-18 00:46:31,350 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.71 sec
2013-09-18 00:46:32,355 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.71 sec
2013-09-18 00:46:33,360 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 2.29 sec
2013-09-18 00:46:34,366 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 2.29 sec
2013-09-18 00:46:35,371 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 2.29 sec
MapReduce Total cumulative CPU time: 2 seconds 290 msec
Ended Job = job_201309172235_0125
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 11 Cumulative CPU: 1333.41 sec HDFS Read: 978257057 HDFS Write: 4324 SUCCESS
Job 1: Map: 1 Reduce: 1 Cumulative CPU: 2.29 sec HDFS Read: 7901 HDFS Write: 790 SUCCESS
Total MapReduce CPU Time Spent: 22 minutes 15 seconds 700 msec
OK
Time taken: 112.58 seconds, Fetched: 25 row(s)
hive> quit;
times: 2
query: SELECT CounterID, avg(length(URL)) AS l, count(*) AS c FROM hits_100m WHERE URL != '' GROUP BY CounterID HAVING count(*) > 100000 ORDER BY l DESC LIMIT 25;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_25268@mturlrep13_201309180046_2058731462.txt
hive> SELECT CounterID, avg(length(URL)) AS l, count(*) AS c FROM hits_100m WHERE URL != '' GROUP BY CounterID HAVING count(*) > 100000 ORDER BY l DESC LIMIT 25;;
Total MapReduce jobs = 2
Launching Job 1 out of 2
Number of reduce tasks not specified. Estimated from input data size: 11
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0126
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 11
2013-09-18 00:46:46,178 Stage-1 map = 0%, reduce = 0%
2013-09-18 00:46:59,296 Stage-1 map = 4%, reduce = 0%
2013-09-18 00:47:00,310 Stage-1 map = 5%, reduce = 0%
2013-09-18 00:47:01,316 Stage-1 map = 6%, reduce = 0%
2013-09-18 00:47:03,326 Stage-1 map = 8%, reduce = 0%
2013-09-18 00:47:04,331 Stage-1 map = 13%, reduce = 0%
2013-09-18 00:47:05,337 Stage-1 map = 16%, reduce = 0%
2013-09-18 00:47:06,343 Stage-1 map = 19%, reduce = 0%
2013-09-18 00:47:07,364 Stage-1 map = 25%, reduce = 0%, Cumulative CPU 14.43 sec
2013-09-18 00:47:08,406 Stage-1 map = 30%, reduce = 0%, Cumulative CPU 14.43 sec
2013-09-18 00:47:09,418 Stage-1 map = 34%, reduce = 0%, Cumulative CPU 14.43 sec
2013-09-18 00:47:10,425 Stage-1 map = 39%, reduce = 0%, Cumulative CPU 14.43 sec
2013-09-18 00:47:11,459 Stage-1 map = 43%, reduce = 0%, Cumulative CPU 14.43 sec
2013-09-18 00:47:12,470 Stage-1 map = 46%, reduce = 0%, Cumulative CPU 14.43 sec
2013-09-18 00:47:13,476 Stage-1 map = 52%, reduce = 0%, Cumulative CPU 14.43 sec
2013-09-18 00:47:14,517 Stage-1 map = 56%, reduce = 0%, Cumulative CPU 14.43 sec
2013-09-18 00:47:15,536 Stage-1 map = 60%, reduce = 0%, Cumulative CPU 14.43 sec
2013-09-18 00:47:16,546 Stage-1 map = 67%, reduce = 0%, Cumulative CPU 14.43 sec
2013-09-18 00:47:17,560 Stage-1 map = 70%, reduce = 0%, Cumulative CPU 14.43 sec
2013-09-18 00:47:18,584 Stage-1 map = 74%, reduce = 0%, Cumulative CPU 14.43 sec
2013-09-18 00:47:19,602 Stage-1 map = 80%, reduce = 0%, Cumulative CPU 14.43 sec
2013-09-18 00:47:20,610 Stage-1 map = 83%, reduce = 0%, Cumulative CPU 40.87 sec
2013-09-18 00:47:21,619 Stage-1 map = 85%, reduce = 0%, Cumulative CPU 69.23 sec
2013-09-18 00:47:22,628 Stage-1 map = 88%, reduce = 0%, Cumulative CPU 237.25 sec
2013-09-18 00:47:23,638 Stage-1 map = 91%, reduce = 0%, Cumulative CPU 408.41 sec
2013-09-18 00:47:24,670 Stage-1 map = 95%, reduce = 0%, Cumulative CPU 645.37 sec
2013-09-18 00:47:25,677 Stage-1 map = 99%, reduce = 0%, Cumulative CPU 955.16 sec
2013-09-18 00:47:26,685 Stage-1 map = 99%, reduce = 0%, Cumulative CPU 987.34 sec
2013-09-18 00:47:27,691 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1085.3 sec
2013-09-18 00:47:28,696 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1085.3 sec
2013-09-18 00:47:29,702 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1085.3 sec
2013-09-18 00:47:30,722 Stage-1 map = 100%, reduce = 4%, Cumulative CPU 1085.3 sec
2013-09-18 00:47:31,729 Stage-1 map = 100%, reduce = 15%, Cumulative CPU 1085.3 sec
2013-09-18 00:47:32,735 Stage-1 map = 100%, reduce = 16%, Cumulative CPU 1085.3 sec
2013-09-18 00:47:33,741 Stage-1 map = 100%, reduce = 17%, Cumulative CPU 1085.3 sec
2013-09-18 00:47:34,747 Stage-1 map = 100%, reduce = 30%, Cumulative CPU 1085.3 sec
2013-09-18 00:47:35,753 Stage-1 map = 100%, reduce = 36%, Cumulative CPU 1085.3 sec
2013-09-18 00:47:36,759 Stage-1 map = 100%, reduce = 40%, Cumulative CPU 1085.3 sec
2013-09-18 00:47:37,765 Stage-1 map = 100%, reduce = 45%, Cumulative CPU 1085.3 sec
2013-09-18 00:47:38,774 Stage-1 map = 100%, reduce = 49%, Cumulative CPU 1196.28 sec
2013-09-18 00:47:39,781 Stage-1 map = 100%, reduce = 51%, Cumulative CPU 1196.28 sec
2013-09-18 00:47:40,787 Stage-1 map = 100%, reduce = 68%, Cumulative CPU 1196.28 sec
2013-09-18 00:47:41,794 Stage-1 map = 100%, reduce = 75%, Cumulative CPU 1196.28 sec
2013-09-18 00:47:42,800 Stage-1 map = 100%, reduce = 76%, Cumulative CPU 1201.93 sec
2013-09-18 00:47:43,806 Stage-1 map = 100%, reduce = 81%, Cumulative CPU 1201.93 sec
2013-09-18 00:47:44,820 Stage-1 map = 100%, reduce = 82%, Cumulative CPU 1209.07 sec
2013-09-18 00:47:45,826 Stage-1 map = 100%, reduce = 82%, Cumulative CPU 1209.07 sec
2013-09-18 00:47:46,832 Stage-1 map = 100%, reduce = 85%, Cumulative CPU 1209.07 sec
2013-09-18 00:47:47,838 Stage-1 map = 100%, reduce = 90%, Cumulative CPU 1220.62 sec
2013-09-18 00:47:48,844 Stage-1 map = 100%, reduce = 90%, Cumulative CPU 1220.62 sec
2013-09-18 00:47:49,850 Stage-1 map = 100%, reduce = 95%, Cumulative CPU 1246.48 sec
2013-09-18 00:47:50,856 Stage-1 map = 100%, reduce = 96%, Cumulative CPU 1260.92 sec
2013-09-18 00:47:51,862 Stage-1 map = 100%, reduce = 97%, Cumulative CPU 1276.65 sec
2013-09-18 00:47:52,868 Stage-1 map = 100%, reduce = 97%, Cumulative CPU 1293.47 sec
2013-09-18 00:47:53,873 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1293.47 sec
2013-09-18 00:47:56,015 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1293.47 sec
2013-09-18 00:47:57,021 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1293.47 sec
2013-09-18 00:47:58,026 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1293.47 sec
2013-09-18 00:47:59,031 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1293.47 sec
2013-09-18 00:48:00,036 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1293.47 sec
2013-09-18 00:48:01,041 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1293.47 sec
2013-09-18 00:48:02,046 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1293.47 sec
2013-09-18 00:48:03,366 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1293.47 sec
2013-09-18 00:48:04,371 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1293.47 sec
2013-09-18 00:48:05,376 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1293.47 sec
2013-09-18 00:48:06,382 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 1328.31 sec
2013-09-18 00:48:09,722 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 1328.31 sec
MapReduce Total cumulative CPU time: 22 minutes 8 seconds 310 msec
Ended Job = job_201309172235_0126
Launching Job 2 out of 2
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0127
Hadoop job information for Stage-2: number of mappers: 1; number of reducers: 1
2013-09-18 00:48:13,466 Stage-2 map = 0%, reduce = 0%
2013-09-18 00:48:15,474 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.82 sec
2013-09-18 00:48:16,478 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.82 sec
2013-09-18 00:48:17,481 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.82 sec
2013-09-18 00:48:18,485 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.82 sec
2013-09-18 00:48:19,489 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.82 sec
2013-09-18 00:48:20,494 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.82 sec
2013-09-18 00:48:21,497 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.82 sec
2013-09-18 00:48:22,502 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 2.38 sec
2013-09-18 00:48:23,508 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 2.38 sec
2013-09-18 00:48:24,513 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 2.38 sec
MapReduce Total cumulative CPU time: 2 seconds 380 msec
Ended Job = job_201309172235_0127
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 11 Cumulative CPU: 1328.31 sec HDFS Read: 978257057 HDFS Write: 4324 SUCCESS
Job 1: Map: 1 Reduce: 1 Cumulative CPU: 2.38 sec HDFS Read: 7901 HDFS Write: 790 SUCCESS
Total MapReduce CPU Time Spent: 22 minutes 10 seconds 690 msec
OK
Time taken: 107.297 seconds, Fetched: 25 row(s)
hive> quit;
times: 3
query: SELECT CounterID, avg(length(URL)) AS l, count(*) AS c FROM hits_100m WHERE URL != '' GROUP BY CounterID HAVING count(*) > 100000 ORDER BY l DESC LIMIT 25;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_32317@mturlrep13_201309180048_1314756439.txt
hive> SELECT CounterID, avg(length(URL)) AS l, count(*) AS c FROM hits_100m WHERE URL != '' GROUP BY CounterID HAVING count(*) > 100000 ORDER BY l DESC LIMIT 25;;
Total MapReduce jobs = 2
Launching Job 1 out of 2
Number of reduce tasks not specified. Estimated from input data size: 11
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0128
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 11
2013-09-18 00:48:35,184 Stage-1 map = 0%, reduce = 0%
2013-09-18 00:48:48,323 Stage-1 map = 5%, reduce = 0%
2013-09-18 00:48:49,329 Stage-1 map = 7%, reduce = 0%
2013-09-18 00:48:53,356 Stage-1 map = 11%, reduce = 0%
2013-09-18 00:48:54,367 Stage-1 map = 15%, reduce = 0%
2013-09-18 00:48:55,377 Stage-1 map = 18%, reduce = 0%
2013-09-18 00:48:56,389 Stage-1 map = 23%, reduce = 0%
2013-09-18 00:48:57,404 Stage-1 map = 29%, reduce = 0%
2013-09-18 00:48:58,435 Stage-1 map = 32%, reduce = 0%
2013-09-18 00:48:59,450 Stage-1 map = 37%, reduce = 0%, Cumulative CPU 15.22 sec
2013-09-18 00:49:00,470 Stage-1 map = 43%, reduce = 0%, Cumulative CPU 15.22 sec
2013-09-18 00:49:01,482 Stage-1 map = 46%, reduce = 0%, Cumulative CPU 15.22 sec
2013-09-18 00:49:02,500 Stage-1 map = 52%, reduce = 0%, Cumulative CPU 15.22 sec
2013-09-18 00:49:03,514 Stage-1 map = 56%, reduce = 0%, Cumulative CPU 15.22 sec
2013-09-18 00:49:04,524 Stage-1 map = 60%, reduce = 0%, Cumulative CPU 15.22 sec
2013-09-18 00:49:05,530 Stage-1 map = 67%, reduce = 0%, Cumulative CPU 15.22 sec
2013-09-18 00:49:06,542 Stage-1 map = 71%, reduce = 0%, Cumulative CPU 15.22 sec
2013-09-18 00:49:07,548 Stage-1 map = 72%, reduce = 0%, Cumulative CPU 15.22 sec
2013-09-18 00:49:08,560 Stage-1 map = 78%, reduce = 0%, Cumulative CPU 15.22 sec
2013-09-18 00:49:09,568 Stage-1 map = 81%, reduce = 0%, Cumulative CPU 15.22 sec
2013-09-18 00:49:10,574 Stage-1 map = 84%, reduce = 0%, Cumulative CPU 69.2 sec
2013-09-18 00:49:11,591 Stage-1 map = 87%, reduce = 0%, Cumulative CPU 154.6 sec
2013-09-18 00:49:12,600 Stage-1 map = 90%, reduce = 0%, Cumulative CPU 386.29 sec
2013-09-18 00:49:13,616 Stage-1 map = 95%, reduce = 0%, Cumulative CPU 684.84 sec
2013-09-18 00:49:14,628 Stage-1 map = 98%, reduce = 0%, Cumulative CPU 904.62 sec
2013-09-18 00:49:15,634 Stage-1 map = 99%, reduce = 0%, Cumulative CPU 999.79 sec
2013-09-18 00:49:16,640 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1097.36 sec
2013-09-18 00:49:17,645 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1097.36 sec
2013-09-18 00:49:18,651 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1097.36 sec
2013-09-18 00:49:19,657 Stage-1 map = 100%, reduce = 2%, Cumulative CPU 1097.36 sec
2013-09-18 00:49:20,664 Stage-1 map = 100%, reduce = 12%, Cumulative CPU 1097.36 sec
2013-09-18 00:49:21,693 Stage-1 map = 100%, reduce = 17%, Cumulative CPU 1097.36 sec
2013-09-18 00:49:22,703 Stage-1 map = 100%, reduce = 18%, Cumulative CPU 1097.36 sec
2013-09-18 00:49:23,709 Stage-1 map = 100%, reduce = 29%, Cumulative CPU 1097.36 sec
2013-09-18 00:49:24,715 Stage-1 map = 100%, reduce = 36%, Cumulative CPU 1097.36 sec
2013-09-18 00:49:25,721 Stage-1 map = 100%, reduce = 36%, Cumulative CPU 1097.36 sec
2013-09-18 00:49:26,728 Stage-1 map = 100%, reduce = 41%, Cumulative CPU 1097.36 sec
2013-09-18 00:49:27,736 Stage-1 map = 100%, reduce = 49%, Cumulative CPU 1108.08 sec
2013-09-18 00:49:28,742 Stage-1 map = 100%, reduce = 49%, Cumulative CPU 1108.08 sec
2013-09-18 00:49:29,748 Stage-1 map = 100%, reduce = 64%, Cumulative CPU 1108.08 sec
2013-09-18 00:49:30,754 Stage-1 map = 100%, reduce = 75%, Cumulative CPU 1108.08 sec
2013-09-18 00:49:31,760 Stage-1 map = 100%, reduce = 77%, Cumulative CPU 1124.02 sec
2013-09-18 00:49:32,766 Stage-1 map = 100%, reduce = 81%, Cumulative CPU 1139.64 sec
2013-09-18 00:49:33,772 Stage-1 map = 100%, reduce = 82%, Cumulative CPU 1139.64 sec
2013-09-18 00:49:34,778 Stage-1 map = 100%, reduce = 82%, Cumulative CPU 1139.64 sec
2013-09-18 00:49:35,783 Stage-1 map = 100%, reduce = 85%, Cumulative CPU 1139.64 sec
2013-09-18 00:49:36,789 Stage-1 map = 100%, reduce = 90%, Cumulative CPU 1160.79 sec
2013-09-18 00:49:37,795 Stage-1 map = 100%, reduce = 90%, Cumulative CPU 1160.79 sec
2013-09-18 00:49:38,801 Stage-1 map = 100%, reduce = 92%, Cumulative CPU 1313.18 sec
2013-09-18 00:49:39,807 Stage-1 map = 100%, reduce = 95%, Cumulative CPU 1314.93 sec
2013-09-18 00:49:40,813 Stage-1 map = 100%, reduce = 96%, Cumulative CPU 1321.52 sec
2013-09-18 00:49:41,819 Stage-1 map = 100%, reduce = 96%, Cumulative CPU 1321.52 sec
2013-09-18 00:49:42,825 Stage-1 map = 100%, reduce = 98%, Cumulative CPU 1332.94 sec
2013-09-18 00:49:43,911 Stage-1 map = 100%, reduce = 98%, Cumulative CPU 1332.94 sec
2013-09-18 00:49:44,917 Stage-1 map = 100%, reduce = 98%, Cumulative CPU 1332.94 sec
2013-09-18 00:49:45,922 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 1332.94 sec
2013-09-18 00:49:46,926 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 1332.94 sec
2013-09-18 00:49:47,931 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 1340.83 sec
2013-09-18 00:49:50,236 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 1340.83 sec
2013-09-18 00:49:51,240 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 1340.83 sec
2013-09-18 00:49:52,245 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 1340.83 sec
2013-09-18 00:49:53,256 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 1340.83 sec
MapReduce Total cumulative CPU time: 22 minutes 20 seconds 830 msec
Ended Job = job_201309172235_0128
Launching Job 2 out of 2
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0129
Hadoop job information for Stage-2: number of mappers: 1; number of reducers: 1
2013-09-18 00:50:01,798 Stage-2 map = 0%, reduce = 0%
2013-09-18 00:50:03,805 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.79 sec
2013-09-18 00:50:04,810 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.79 sec
2013-09-18 00:50:05,814 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.79 sec
2013-09-18 00:50:06,819 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.79 sec
2013-09-18 00:50:07,823 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.79 sec
2013-09-18 00:50:08,828 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.79 sec
2013-09-18 00:50:09,832 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.79 sec
2013-09-18 00:50:10,837 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 0.79 sec
2013-09-18 00:50:11,843 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 2.4 sec
2013-09-18 00:50:12,847 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 2.4 sec
MapReduce Total cumulative CPU time: 2 seconds 400 msec
Ended Job = job_201309172235_0129
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 11 Cumulative CPU: 1340.83 sec HDFS Read: 978257057 HDFS Write: 4324 SUCCESS
Job 1: Map: 1 Reduce: 1 Cumulative CPU: 2.4 sec HDFS Read: 7901 HDFS Write: 790 SUCCESS
Total MapReduce CPU Time Spent: 22 minutes 23 seconds 230 msec
OK
Time taken: 106.493 seconds, Fetched: 25 row(s)
hive> quit;
-- считаем средние длины URL для крупных счётчиков.;
times: 1
query: SELECT SUBSTRING(SUBSTRING(Referer, FIND_IN_SET('//', Referer) + 2), 1, GREATEST(0, FIND_IN_SET('/', SUBSTRING(Referer, FIND_IN_SET('//', Referer) + 2)) - 1)), avg(length(Referer)) AS l, count(*) AS c, MAX(Referer) FROM hits_100m WHERE Referer != '' GROUP BY SUBSTRING(SUBSTRING(Referer, FIND_IN_SET('//', Referer) + 2), 1, GREATEST(0, FIND_IN_SET('/', SUBSTRING(Referer, FIND_IN_SET('//', Referer) + 2)) - 1)) HAVING count(*) > 100000 ORDER BY l DESC LIMIT 25;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_8110@mturlrep13_201309180050_337201518.txt
hive> SELECT SUBSTRING(SUBSTRING(Referer, FIND_IN_SET('//', Referer) + 2), 1, GREATEST(0, FIND_IN_SET('/', SUBSTRING(Referer, FIND_IN_SET('//', Referer) + 2)) - 1)), avg(length(Referer)) AS l, count(*) AS c, MAX(Referer) FROM hits_100m WHERE Referer != '' GROUP BY SUBSTRING(SUBSTRING(Referer, FIND_IN_SET('//', Referer) + 2), 1, GREATEST(0, FIND_IN_SET('/', SUBSTRING(Referer, FIND_IN_SET('//', Referer) + 2)) - 1)) HAVING count(*) > 100000 ORDER BY l DESC LIMIT 25;;
FAILED: SemanticException [Error 10011]: Line 1:337 Invalid function 'GREATEST'
hive> quit;
times: 2
query: SELECT SUBSTRING(SUBSTRING(Referer, FIND_IN_SET('//', Referer) + 2), 1, GREATEST(0, FIND_IN_SET('/', SUBSTRING(Referer, FIND_IN_SET('//', Referer) + 2)) - 1)), avg(length(Referer)) AS l, count(*) AS c, MAX(Referer) FROM hits_100m WHERE Referer != '' GROUP BY SUBSTRING(SUBSTRING(Referer, FIND_IN_SET('//', Referer) + 2), 1, GREATEST(0, FIND_IN_SET('/', SUBSTRING(Referer, FIND_IN_SET('//', Referer) + 2)) - 1)) HAVING count(*) > 100000 ORDER BY l DESC LIMIT 25;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_8329@mturlrep13_201309180050_1201800197.txt
hive> SELECT SUBSTRING(SUBSTRING(Referer, FIND_IN_SET('//', Referer) + 2), 1, GREATEST(0, FIND_IN_SET('/', SUBSTRING(Referer, FIND_IN_SET('//', Referer) + 2)) - 1)), avg(length(Referer)) AS l, count(*) AS c, MAX(Referer) FROM hits_100m WHERE Referer != '' GROUP BY SUBSTRING(SUBSTRING(Referer, FIND_IN_SET('//', Referer) + 2), 1, GREATEST(0, FIND_IN_SET('/', SUBSTRING(Referer, FIND_IN_SET('//', Referer) + 2)) - 1)) HAVING count(*) > 100000 ORDER BY l DESC LIMIT 25;;
FAILED: SemanticException [Error 10011]: Line 1:337 Invalid function 'GREATEST'
hive> quit;
times: 3
query: SELECT SUBSTRING(SUBSTRING(Referer, FIND_IN_SET('//', Referer) + 2), 1, GREATEST(0, FIND_IN_SET('/', SUBSTRING(Referer, FIND_IN_SET('//', Referer) + 2)) - 1)), avg(length(Referer)) AS l, count(*) AS c, MAX(Referer) FROM hits_100m WHERE Referer != '' GROUP BY SUBSTRING(SUBSTRING(Referer, FIND_IN_SET('//', Referer) + 2), 1, GREATEST(0, FIND_IN_SET('/', SUBSTRING(Referer, FIND_IN_SET('//', Referer) + 2)) - 1)) HAVING count(*) > 100000 ORDER BY l DESC LIMIT 25;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_8542@mturlrep13_201309180050_468390903.txt
hive> SELECT SUBSTRING(SUBSTRING(Referer, FIND_IN_SET('//', Referer) + 2), 1, GREATEST(0, FIND_IN_SET('/', SUBSTRING(Referer, FIND_IN_SET('//', Referer) + 2)) - 1)), avg(length(Referer)) AS l, count(*) AS c, MAX(Referer) FROM hits_100m WHERE Referer != '' GROUP BY SUBSTRING(SUBSTRING(Referer, FIND_IN_SET('//', Referer) + 2), 1, GREATEST(0, FIND_IN_SET('/', SUBSTRING(Referer, FIND_IN_SET('//', Referer) + 2)) - 1)) HAVING count(*) > 100000 ORDER BY l DESC LIMIT 25;;
FAILED: SemanticException [Error 10011]: Line 1:337 Invalid function 'GREATEST'
hive> quit;
-- то же самое, но с разбивкой по доменам.;
times: 1
query: SELECT sum(ResolutionWidth), sum(ResolutionWidth + 1), sum(ResolutionWidth + 2), sum(ResolutionWidth + 3), sum(ResolutionWidth + 4), sum(ResolutionWidth + 5), sum(ResolutionWidth + 6), sum(ResolutionWidth + 7), sum(ResolutionWidth + 8), sum(ResolutionWidth + 9), sum(ResolutionWidth + 10), sum(ResolutionWidth + 11), sum(ResolutionWidth + 12), sum(ResolutionWidth + 13), sum(ResolutionWidth + 14), sum(ResolutionWidth + 15), sum(ResolutionWidth + 16), sum(ResolutionWidth + 17), sum(ResolutionWidth + 18), sum(ResolutionWidth + 19), sum(ResolutionWidth + 20), sum(ResolutionWidth + 21), sum(ResolutionWidth + 22), sum(ResolutionWidth + 23), sum(ResolutionWidth + 24), sum(ResolutionWidth + 25), sum(ResolutionWidth + 26), sum(ResolutionWidth + 27), sum(ResolutionWidth + 28), sum(ResolutionWidth + 29), sum(ResolutionWidth + 30), sum(ResolutionWidth + 31), sum(ResolutionWidth + 32), sum(ResolutionWidth + 33), sum(ResolutionWidth + 34), sum(ResolutionWidth + 35), sum(ResolutionWidth + 36), sum(ResolutionWidth + 37), sum(ResolutionWidth + 38), sum(ResolutionWidth + 39), sum(ResolutionWidth + 40), sum(ResolutionWidth + 41), sum(ResolutionWidth + 42), sum(ResolutionWidth + 43), sum(ResolutionWidth + 44), sum(ResolutionWidth + 45), sum(ResolutionWidth + 46), sum(ResolutionWidth + 47), sum(ResolutionWidth + 48), sum(ResolutionWidth + 49), sum(ResolutionWidth + 50), sum(ResolutionWidth + 51), sum(ResolutionWidth + 52), sum(ResolutionWidth + 53), sum(ResolutionWidth + 54), sum(ResolutionWidth + 55), sum(ResolutionWidth + 56), sum(ResolutionWidth + 57), sum(ResolutionWidth + 58), sum(ResolutionWidth + 59), sum(ResolutionWidth + 60), sum(ResolutionWidth + 61), sum(ResolutionWidth + 62), sum(ResolutionWidth + 63), sum(ResolutionWidth + 64), sum(ResolutionWidth + 65), sum(ResolutionWidth + 66), sum(ResolutionWidth + 67), sum(ResolutionWidth + 68), sum(ResolutionWidth + 69), sum(ResolutionWidth + 70), sum(ResolutionWidth + 71), sum(ResolutionWidth + 72), sum(ResolutionWidth + 73), sum(ResolutionWidth + 74), sum(ResolutionWidth + 75), sum(ResolutionWidth + 76), sum(ResolutionWidth + 77), sum(ResolutionWidth + 78), sum(ResolutionWidth + 79), sum(ResolutionWidth + 80), sum(ResolutionWidth + 81), sum(ResolutionWidth + 82), sum(ResolutionWidth + 83), sum(ResolutionWidth + 84), sum(ResolutionWidth + 85), sum(ResolutionWidth + 86), sum(ResolutionWidth + 87), sum(ResolutionWidth + 88), sum(ResolutionWidth + 89) FROM hits_100m;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_8757@mturlrep13_201309180050_2105633958.txt
hive> SELECT sum(ResolutionWidth), sum(ResolutionWidth + 1), sum(ResolutionWidth + 2), sum(ResolutionWidth + 3), sum(ResolutionWidth + 4), sum(ResolutionWidth + 5), sum(ResolutionWidth + 6), sum(ResolutionWidth + 7), sum(ResolutionWidth + 8), sum(ResolutionWidth + 9), sum(ResolutionWidth + 10), sum(ResolutionWidth + 11), sum(ResolutionWidth + 12), sum(ResolutionWidth + 13), sum(ResolutionWidth + 14), sum(ResolutionWidth + 15), sum(ResolutionWidth + 16), sum(ResolutionWidth + 17), sum(ResolutionWidth + 18), sum(ResolutionWidth + 19), sum(ResolutionWidth + 20), sum(ResolutionWidth + 21), sum(ResolutionWidth + 22), sum(ResolutionWidth + 23), sum(ResolutionWidth + 24), sum(ResolutionWidth + 25), sum(ResolutionWidth + 26), sum(ResolutionWidth + 27), sum(ResolutionWidth + 28), sum(ResolutionWidth + 29), sum(ResolutionWidth + 30), sum(ResolutionWidth + 31), sum(ResolutionWidth + 32), sum(ResolutionWidth + 33), sum(ResolutionWidth + 34), sum(ResolutionWidth + 35), sum(ResolutionWidth + 36), sum(ResolutionWidth + 37), sum(ResolutionWidth + 38), sum(ResolutionWidth + 39), sum(ResolutionWidth + 40), sum(ResolutionWidth + 41), sum(ResolutionWidth + 42), sum(ResolutionWidth + 43), sum(ResolutionWidth + 44), sum(ResolutionWidth + 45), sum(ResolutionWidth + 46), sum(ResolutionWidth + 47), sum(ResolutionWidth + 48), sum(ResolutionWidth + 49), sum(ResolutionWidth + 50), sum(ResolutionWidth + 51), sum(ResolutionWidth + 52), sum(ResolutionWidth + 53), sum(ResolutionWidth + 54), sum(ResolutionWidth + 55), sum(ResolutionWidth + 56), sum(ResolutionWidth + 57), sum(ResolutionWidth + 58), sum(ResolutionWidth + 59), sum(ResolutionWidth + 60), sum(ResolutionWidth + 61), sum(ResolutionWidth + 62), sum(ResolutionWidth + 63), sum(ResolutionWidth + 64), sum(ResolutionWidth + 65), sum(ResolutionWidth + 66), sum(ResolutionWidth + 67), sum(ResolutionWidth + 68), sum(ResolutionWidth + 69), sum(ResolutionWidth + 70), sum(ResolutionWidth + 71), sum(ResolutionWidth + 72), sum(ResolutionWidth + 73), sum(ResolutionWidth + 74), sum(ResolutionWidth + 75), sum(ResolutionWidth + 76), sum(ResolutionWidth + 77), sum(ResolutionWidth + 78), sum(ResolutionWidth + 79), sum(ResolutionWidth + 80), sum(ResolutionWidth + 81), sum(ResolutionWidth + 82), sum(ResolutionWidth + 83), sum(ResolutionWidth + 84), sum(ResolutionWidth + 85), sum(ResolutionWidth + 86), sum(ResolutionWidth + 87), sum(ResolutionWidth + 88), sum(ResolutionWidth + 89) FROM hits_100m;;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0130
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 1
2013-09-18 00:50:56,035 Stage-1 map = 0%, reduce = 0%
2013-09-18 00:51:21,228 Stage-1 map = 1%, reduce = 0%
2013-09-18 00:51:22,233 Stage-1 map = 4%, reduce = 0%
2013-09-18 00:51:23,242 Stage-1 map = 5%, reduce = 0%
2013-09-18 00:51:24,247 Stage-1 map = 6%, reduce = 0%
2013-09-18 00:51:25,251 Stage-1 map = 11%, reduce = 0%
2013-09-18 00:51:26,256 Stage-1 map = 13%, reduce = 0%
2013-09-18 00:51:28,265 Stage-1 map = 14%, reduce = 0%
2013-09-18 00:51:34,323 Stage-1 map = 17%, reduce = 0%
2013-09-18 00:51:35,333 Stage-1 map = 19%, reduce = 0%
2013-09-18 00:51:36,342 Stage-1 map = 20%, reduce = 0%
2013-09-18 00:51:37,347 Stage-1 map = 21%, reduce = 0%
2013-09-18 00:51:38,352 Stage-1 map = 25%, reduce = 0%
2013-09-18 00:51:39,363 Stage-1 map = 26%, reduce = 0%, Cumulative CPU 1178.1 sec
2013-09-18 00:51:40,382 Stage-1 map = 26%, reduce = 0%, Cumulative CPU 1178.1 sec
2013-09-18 00:51:41,395 Stage-1 map = 27%, reduce = 0%, Cumulative CPU 1178.1 sec
2013-09-18 00:51:42,410 Stage-1 map = 27%, reduce = 0%, Cumulative CPU 1178.1 sec
2013-09-18 00:51:43,416 Stage-1 map = 29%, reduce = 0%, Cumulative CPU 1178.1 sec
2013-09-18 00:51:44,421 Stage-1 map = 31%, reduce = 0%, Cumulative CPU 1178.1 sec
2013-09-18 00:51:45,427 Stage-1 map = 32%, reduce = 0%, Cumulative CPU 1178.1 sec
2013-09-18 00:51:46,437 Stage-1 map = 32%, reduce = 0%, Cumulative CPU 1178.1 sec
2013-09-18 00:51:47,444 Stage-1 map = 33%, reduce = 0%, Cumulative CPU 1178.1 sec
2013-09-18 00:51:48,450 Stage-1 map = 33%, reduce = 0%, Cumulative CPU 1178.1 sec
2013-09-18 00:51:49,455 Stage-1 map = 34%, reduce = 0%, Cumulative CPU 1178.1 sec
2013-09-18 00:51:50,461 Stage-1 map = 36%, reduce = 0%, Cumulative CPU 1178.1 sec
2013-09-18 00:51:51,466 Stage-1 map = 39%, reduce = 0%, Cumulative CPU 1178.1 sec
2013-09-18 00:51:52,472 Stage-1 map = 39%, reduce = 0%, Cumulative CPU 1178.1 sec
2013-09-18 00:51:53,478 Stage-1 map = 41%, reduce = 0%, Cumulative CPU 1178.1 sec
2013-09-18 00:51:54,483 Stage-1 map = 42%, reduce = 0%, Cumulative CPU 1178.1 sec
2013-09-18 00:51:55,489 Stage-1 map = 43%, reduce = 0%, Cumulative CPU 1178.1 sec
2013-09-18 00:51:56,494 Stage-1 map = 44%, reduce = 0%, Cumulative CPU 1178.1 sec
2013-09-18 00:51:57,500 Stage-1 map = 46%, reduce = 0%, Cumulative CPU 1178.1 sec
2013-09-18 00:51:58,535 Stage-1 map = 46%, reduce = 0%, Cumulative CPU 1178.1 sec
2013-09-18 00:51:59,548 Stage-1 map = 47%, reduce = 0%, Cumulative CPU 1178.1 sec
2013-09-18 00:52:00,553 Stage-1 map = 48%, reduce = 0%, Cumulative CPU 1178.1 sec
2013-09-18 00:52:01,575 Stage-1 map = 48%, reduce = 0%, Cumulative CPU 1178.1 sec
2013-09-18 00:52:02,586 Stage-1 map = 48%, reduce = 0%, Cumulative CPU 1178.1 sec
2013-09-18 00:52:03,591 Stage-1 map = 52%, reduce = 0%, Cumulative CPU 1178.1 sec
2013-09-18 00:52:04,597 Stage-1 map = 52%, reduce = 0%, Cumulative CPU 1178.1 sec
2013-09-18 00:52:05,602 Stage-1 map = 53%, reduce = 0%, Cumulative CPU 1178.1 sec
2013-09-18 00:52:06,608 Stage-1 map = 56%, reduce = 0%, Cumulative CPU 1178.1 sec
2013-09-18 00:52:07,614 Stage-1 map = 57%, reduce = 0%, Cumulative CPU 1178.1 sec
2013-09-18 00:52:08,653 Stage-1 map = 59%, reduce = 0%, Cumulative CPU 1178.1 sec
2013-09-18 00:52:09,659 Stage-1 map = 60%, reduce = 0%, Cumulative CPU 1178.1 sec
2013-09-18 00:52:10,687 Stage-1 map = 61%, reduce = 0%, Cumulative CPU 1178.1 sec
2013-09-18 00:52:11,702 Stage-1 map = 61%, reduce = 0%, Cumulative CPU 1178.1 sec
2013-09-18 00:52:13,057 Stage-1 map = 63%, reduce = 0%, Cumulative CPU 1178.1 sec
2013-09-18 00:52:14,078 Stage-1 map = 63%, reduce = 0%, Cumulative CPU 1178.1 sec
2013-09-18 00:52:15,103 Stage-1 map = 64%, reduce = 0%, Cumulative CPU 1178.1 sec
2013-09-18 00:52:16,131 Stage-1 map = 67%, reduce = 0%, Cumulative CPU 1178.1 sec
2013-09-18 00:52:17,155 Stage-1 map = 67%, reduce = 0%, Cumulative CPU 1178.1 sec
2013-09-18 00:52:18,174 Stage-1 map = 69%, reduce = 0%, Cumulative CPU 1178.1 sec
2013-09-18 00:52:19,198 Stage-1 map = 70%, reduce = 0%, Cumulative CPU 1178.1 sec
2013-09-18 00:52:20,210 Stage-1 map = 70%, reduce = 0%, Cumulative CPU 1212.03 sec
2013-09-18 00:52:21,248 Stage-1 map = 72%, reduce = 0%, Cumulative CPU 1212.03 sec
2013-09-18 00:52:22,262 Stage-1 map = 73%, reduce = 0%, Cumulative CPU 1212.03 sec
2013-09-18 00:52:23,274 Stage-1 map = 74%, reduce = 0%, Cumulative CPU 1212.03 sec
2013-09-18 00:52:24,280 Stage-1 map = 75%, reduce = 0%, Cumulative CPU 1212.03 sec
2013-09-18 00:52:25,290 Stage-1 map = 78%, reduce = 0%, Cumulative CPU 1212.03 sec
2013-09-18 00:52:26,311 Stage-1 map = 78%, reduce = 0%, Cumulative CPU 1212.03 sec
2013-09-18 00:52:27,322 Stage-1 map = 79%, reduce = 0%, Cumulative CPU 1257.13 sec
2013-09-18 00:52:28,327 Stage-1 map = 81%, reduce = 0%, Cumulative CPU 1347.07 sec
2013-09-18 00:52:29,333 Stage-1 map = 82%, reduce = 0%, Cumulative CPU 1347.07 sec
2013-09-18 00:52:30,346 Stage-1 map = 83%, reduce = 0%, Cumulative CPU 1393.25 sec
2013-09-18 00:52:31,358 Stage-1 map = 85%, reduce = 0%, Cumulative CPU 1483.18 sec
2013-09-18 00:52:32,373 Stage-1 map = 87%, reduce = 0%, Cumulative CPU 1674.0 sec
2013-09-18 00:52:33,378 Stage-1 map = 88%, reduce = 0%, Cumulative CPU 1768.61 sec
2013-09-18 00:52:34,383 Stage-1 map = 89%, reduce = 0%, Cumulative CPU 1815.43 sec
2013-09-18 00:52:35,389 Stage-1 map = 90%, reduce = 0%, Cumulative CPU 1865.46 sec
2013-09-18 00:52:36,395 Stage-1 map = 92%, reduce = 0%, Cumulative CPU 2120.58 sec
2013-09-18 00:52:37,401 Stage-1 map = 94%, reduce = 0%, Cumulative CPU 2333.45 sec
2013-09-18 00:52:38,406 Stage-1 map = 94%, reduce = 0%, Cumulative CPU 2333.45 sec
2013-09-18 00:52:39,412 Stage-1 map = 95%, reduce = 0%, Cumulative CPU 3023.48 sec
2013-09-18 00:52:40,418 Stage-1 map = 96%, reduce = 0%, Cumulative CPU 3027.43 sec
2013-09-18 00:52:41,424 Stage-1 map = 97%, reduce = 0%, Cumulative CPU 3039.12 sec
2013-09-18 00:52:42,429 Stage-1 map = 98%, reduce = 0%, Cumulative CPU 3047.25 sec
2013-09-18 00:52:43,434 Stage-1 map = 98%, reduce = 25%, Cumulative CPU 3052.91 sec
2013-09-18 00:52:44,440 Stage-1 map = 99%, reduce = 25%, Cumulative CPU 3052.91 sec
2013-09-18 00:52:45,453 Stage-1 map = 99%, reduce = 25%, Cumulative CPU 3052.91 sec
2013-09-18 00:52:46,459 Stage-1 map = 99%, reduce = 25%, Cumulative CPU 3061.93 sec
2013-09-18 00:52:47,464 Stage-1 map = 99%, reduce = 25%, Cumulative CPU 3061.93 sec
2013-09-18 00:52:48,469 Stage-1 map = 99%, reduce = 25%, Cumulative CPU 3061.93 sec
2013-09-18 00:52:49,474 Stage-1 map = 99%, reduce = 25%, Cumulative CPU 3061.93 sec
2013-09-18 00:52:50,480 Stage-1 map = 99%, reduce = 25%, Cumulative CPU 3061.93 sec
2013-09-18 00:52:51,485 Stage-1 map = 99%, reduce = 25%, Cumulative CPU 3061.93 sec
2013-09-18 00:52:52,490 Stage-1 map = 100%, reduce = 31%, Cumulative CPU 3091.79 sec
2013-09-18 00:52:53,495 Stage-1 map = 100%, reduce = 31%, Cumulative CPU 3091.79 sec
2013-09-18 00:52:54,500 Stage-1 map = 100%, reduce = 31%, Cumulative CPU 3091.79 sec
2013-09-18 00:52:55,504 Stage-1 map = 100%, reduce = 31%, Cumulative CPU 3091.79 sec
2013-09-18 00:52:56,509 Stage-1 map = 100%, reduce = 31%, Cumulative CPU 3091.79 sec
2013-09-18 00:52:57,514 Stage-1 map = 100%, reduce = 31%, Cumulative CPU 3091.79 sec
2013-09-18 00:52:58,521 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 3095.82 sec
2013-09-18 00:52:59,529 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 3095.82 sec
2013-09-18 00:53:00,534 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 3095.82 sec
MapReduce Total cumulative CPU time: 51 minutes 35 seconds 820 msec
Ended Job = job_201309172235_0130
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 1 Cumulative CPU: 3095.82 sec HDFS Read: 77419054 HDFS Write: 1170 SUCCESS
Total MapReduce CPU Time Spent: 51 minutes 35 seconds 820 msec
OK
Time taken: 136.36 seconds, Fetched: 1 row(s)
hive> quit;
times: 2
query: SELECT sum(ResolutionWidth), sum(ResolutionWidth + 1), sum(ResolutionWidth + 2), sum(ResolutionWidth + 3), sum(ResolutionWidth + 4), sum(ResolutionWidth + 5), sum(ResolutionWidth + 6), sum(ResolutionWidth + 7), sum(ResolutionWidth + 8), sum(ResolutionWidth + 9), sum(ResolutionWidth + 10), sum(ResolutionWidth + 11), sum(ResolutionWidth + 12), sum(ResolutionWidth + 13), sum(ResolutionWidth + 14), sum(ResolutionWidth + 15), sum(ResolutionWidth + 16), sum(ResolutionWidth + 17), sum(ResolutionWidth + 18), sum(ResolutionWidth + 19), sum(ResolutionWidth + 20), sum(ResolutionWidth + 21), sum(ResolutionWidth + 22), sum(ResolutionWidth + 23), sum(ResolutionWidth + 24), sum(ResolutionWidth + 25), sum(ResolutionWidth + 26), sum(ResolutionWidth + 27), sum(ResolutionWidth + 28), sum(ResolutionWidth + 29), sum(ResolutionWidth + 30), sum(ResolutionWidth + 31), sum(ResolutionWidth + 32), sum(ResolutionWidth + 33), sum(ResolutionWidth + 34), sum(ResolutionWidth + 35), sum(ResolutionWidth + 36), sum(ResolutionWidth + 37), sum(ResolutionWidth + 38), sum(ResolutionWidth + 39), sum(ResolutionWidth + 40), sum(ResolutionWidth + 41), sum(ResolutionWidth + 42), sum(ResolutionWidth + 43), sum(ResolutionWidth + 44), sum(ResolutionWidth + 45), sum(ResolutionWidth + 46), sum(ResolutionWidth + 47), sum(ResolutionWidth + 48), sum(ResolutionWidth + 49), sum(ResolutionWidth + 50), sum(ResolutionWidth + 51), sum(ResolutionWidth + 52), sum(ResolutionWidth + 53), sum(ResolutionWidth + 54), sum(ResolutionWidth + 55), sum(ResolutionWidth + 56), sum(ResolutionWidth + 57), sum(ResolutionWidth + 58), sum(ResolutionWidth + 59), sum(ResolutionWidth + 60), sum(ResolutionWidth + 61), sum(ResolutionWidth + 62), sum(ResolutionWidth + 63), sum(ResolutionWidth + 64), sum(ResolutionWidth + 65), sum(ResolutionWidth + 66), sum(ResolutionWidth + 67), sum(ResolutionWidth + 68), sum(ResolutionWidth + 69), sum(ResolutionWidth + 70), sum(ResolutionWidth + 71), sum(ResolutionWidth + 72), sum(ResolutionWidth + 73), sum(ResolutionWidth + 74), sum(ResolutionWidth + 75), sum(ResolutionWidth + 76), sum(ResolutionWidth + 77), sum(ResolutionWidth + 78), sum(ResolutionWidth + 79), sum(ResolutionWidth + 80), sum(ResolutionWidth + 81), sum(ResolutionWidth + 82), sum(ResolutionWidth + 83), sum(ResolutionWidth + 84), sum(ResolutionWidth + 85), sum(ResolutionWidth + 86), sum(ResolutionWidth + 87), sum(ResolutionWidth + 88), sum(ResolutionWidth + 89) FROM hits_100m;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_13963@mturlrep13_201309180053_177355133.txt
hive> SELECT sum(ResolutionWidth), sum(ResolutionWidth + 1), sum(ResolutionWidth + 2), sum(ResolutionWidth + 3), sum(ResolutionWidth + 4), sum(ResolutionWidth + 5), sum(ResolutionWidth + 6), sum(ResolutionWidth + 7), sum(ResolutionWidth + 8), sum(ResolutionWidth + 9), sum(ResolutionWidth + 10), sum(ResolutionWidth + 11), sum(ResolutionWidth + 12), sum(ResolutionWidth + 13), sum(ResolutionWidth + 14), sum(ResolutionWidth + 15), sum(ResolutionWidth + 16), sum(ResolutionWidth + 17), sum(ResolutionWidth + 18), sum(ResolutionWidth + 19), sum(ResolutionWidth + 20), sum(ResolutionWidth + 21), sum(ResolutionWidth + 22), sum(ResolutionWidth + 23), sum(ResolutionWidth + 24), sum(ResolutionWidth + 25), sum(ResolutionWidth + 26), sum(ResolutionWidth + 27), sum(ResolutionWidth + 28), sum(ResolutionWidth + 29), sum(ResolutionWidth + 30), sum(ResolutionWidth + 31), sum(ResolutionWidth + 32), sum(ResolutionWidth + 33), sum(ResolutionWidth + 34), sum(ResolutionWidth + 35), sum(ResolutionWidth + 36), sum(ResolutionWidth + 37), sum(ResolutionWidth + 38), sum(ResolutionWidth + 39), sum(ResolutionWidth + 40), sum(ResolutionWidth + 41), sum(ResolutionWidth + 42), sum(ResolutionWidth + 43), sum(ResolutionWidth + 44), sum(ResolutionWidth + 45), sum(ResolutionWidth + 46), sum(ResolutionWidth + 47), sum(ResolutionWidth + 48), sum(ResolutionWidth + 49), sum(ResolutionWidth + 50), sum(ResolutionWidth + 51), sum(ResolutionWidth + 52), sum(ResolutionWidth + 53), sum(ResolutionWidth + 54), sum(ResolutionWidth + 55), sum(ResolutionWidth + 56), sum(ResolutionWidth + 57), sum(ResolutionWidth + 58), sum(ResolutionWidth + 59), sum(ResolutionWidth + 60), sum(ResolutionWidth + 61), sum(ResolutionWidth + 62), sum(ResolutionWidth + 63), sum(ResolutionWidth + 64), sum(ResolutionWidth + 65), sum(ResolutionWidth + 66), sum(ResolutionWidth + 67), sum(ResolutionWidth + 68), sum(ResolutionWidth + 69), sum(ResolutionWidth + 70), sum(ResolutionWidth + 71), sum(ResolutionWidth + 72), sum(ResolutionWidth + 73), sum(ResolutionWidth + 74), sum(ResolutionWidth + 75), sum(ResolutionWidth + 76), sum(ResolutionWidth + 77), sum(ResolutionWidth + 78), sum(ResolutionWidth + 79), sum(ResolutionWidth + 80), sum(ResolutionWidth + 81), sum(ResolutionWidth + 82), sum(ResolutionWidth + 83), sum(ResolutionWidth + 84), sum(ResolutionWidth + 85), sum(ResolutionWidth + 86), sum(ResolutionWidth + 87), sum(ResolutionWidth + 88), sum(ResolutionWidth + 89) FROM hits_100m;;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0131
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 1
2013-09-18 00:53:11,219 Stage-1 map = 0%, reduce = 0%
2013-09-18 00:53:36,459 Stage-1 map = 2%, reduce = 0%
2013-09-18 00:53:38,478 Stage-1 map = 4%, reduce = 0%
2013-09-18 00:53:39,508 Stage-1 map = 7%, reduce = 0%, Cumulative CPU 679.85 sec
2013-09-18 00:53:40,526 Stage-1 map = 10%, reduce = 0%, Cumulative CPU 679.85 sec
2013-09-18 00:53:41,534 Stage-1 map = 13%, reduce = 0%, Cumulative CPU 679.85 sec
2013-09-18 00:53:42,546 Stage-1 map = 13%, reduce = 0%, Cumulative CPU 679.85 sec
2013-09-18 00:53:43,564 Stage-1 map = 13%, reduce = 0%, Cumulative CPU 679.85 sec
2013-09-18 00:53:44,570 Stage-1 map = 14%, reduce = 0%, Cumulative CPU 679.85 sec
2013-09-18 00:53:45,576 Stage-1 map = 14%, reduce = 0%, Cumulative CPU 679.85 sec
2013-09-18 00:53:46,582 Stage-1 map = 14%, reduce = 0%, Cumulative CPU 679.85 sec
2013-09-18 00:53:47,588 Stage-1 map = 15%, reduce = 0%, Cumulative CPU 679.85 sec
2013-09-18 00:53:48,594 Stage-1 map = 15%, reduce = 0%, Cumulative CPU 679.85 sec
2013-09-18 00:53:49,600 Stage-1 map = 17%, reduce = 0%, Cumulative CPU 679.85 sec
2013-09-18 00:53:50,610 Stage-1 map = 20%, reduce = 0%, Cumulative CPU 679.85 sec
2013-09-18 00:53:51,667 Stage-1 map = 22%, reduce = 0%, Cumulative CPU 679.85 sec
2013-09-18 00:53:52,674 Stage-1 map = 23%, reduce = 0%, Cumulative CPU 679.85 sec
2013-09-18 00:53:53,680 Stage-1 map = 25%, reduce = 0%, Cumulative CPU 679.85 sec
2013-09-18 00:53:54,685 Stage-1 map = 25%, reduce = 0%, Cumulative CPU 679.85 sec
2013-09-18 00:53:55,691 Stage-1 map = 25%, reduce = 0%, Cumulative CPU 679.85 sec
2013-09-18 00:53:56,703 Stage-1 map = 27%, reduce = 0%, Cumulative CPU 679.85 sec
2013-09-18 00:53:57,708 Stage-1 map = 28%, reduce = 0%, Cumulative CPU 679.85 sec
2013-09-18 00:53:58,720 Stage-1 map = 29%, reduce = 0%, Cumulative CPU 679.85 sec
2013-09-18 00:53:59,732 Stage-1 map = 30%, reduce = 0%, Cumulative CPU 679.85 sec
2013-09-18 00:54:00,738 Stage-1 map = 31%, reduce = 0%, Cumulative CPU 679.85 sec
2013-09-18 00:54:01,743 Stage-1 map = 32%, reduce = 0%, Cumulative CPU 679.85 sec
2013-09-18 00:54:02,749 Stage-1 map = 32%, reduce = 0%, Cumulative CPU 679.85 sec
2013-09-18 00:54:03,754 Stage-1 map = 34%, reduce = 0%, Cumulative CPU 679.85 sec
2013-09-18 00:54:04,759 Stage-1 map = 36%, reduce = 0%, Cumulative CPU 679.85 sec
2013-09-18 00:54:05,766 Stage-1 map = 37%, reduce = 0%, Cumulative CPU 679.85 sec
2013-09-18 00:54:06,771 Stage-1 map = 38%, reduce = 0%, Cumulative CPU 679.85 sec
2013-09-18 00:54:07,784 Stage-1 map = 39%, reduce = 0%, Cumulative CPU 679.85 sec
2013-09-18 00:54:08,789 Stage-1 map = 40%, reduce = 0%, Cumulative CPU 679.85 sec
2013-09-18 00:54:09,795 Stage-1 map = 42%, reduce = 0%, Cumulative CPU 679.85 sec
2013-09-18 00:54:10,800 Stage-1 map = 43%, reduce = 0%, Cumulative CPU 679.85 sec
2013-09-18 00:54:11,806 Stage-1 map = 44%, reduce = 0%, Cumulative CPU 679.85 sec
2013-09-18 00:54:12,811 Stage-1 map = 46%, reduce = 0%, Cumulative CPU 679.85 sec
2013-09-18 00:54:13,823 Stage-1 map = 46%, reduce = 0%, Cumulative CPU 679.85 sec
2013-09-18 00:54:14,829 Stage-1 map = 48%, reduce = 0%, Cumulative CPU 679.85 sec
2013-09-18 00:54:15,834 Stage-1 map = 49%, reduce = 0%, Cumulative CPU 679.85 sec
2013-09-18 00:54:16,844 Stage-1 map = 50%, reduce = 0%, Cumulative CPU 679.85 sec
2013-09-18 00:54:17,850 Stage-1 map = 51%, reduce = 0%, Cumulative CPU 679.85 sec
2013-09-18 00:54:18,858 Stage-1 map = 52%, reduce = 0%, Cumulative CPU 679.85 sec
2013-09-18 00:54:19,871 Stage-1 map = 53%, reduce = 0%, Cumulative CPU 679.85 sec
2013-09-18 00:54:20,878 Stage-1 map = 55%, reduce = 0%, Cumulative CPU 679.85 sec
2013-09-18 00:54:21,883 Stage-1 map = 56%, reduce = 0%, Cumulative CPU 679.85 sec
2013-09-18 00:54:22,888 Stage-1 map = 57%, reduce = 0%, Cumulative CPU 679.85 sec
2013-09-18 00:54:23,894 Stage-1 map = 58%, reduce = 0%, Cumulative CPU 679.85 sec
2013-09-18 00:54:24,901 Stage-1 map = 59%, reduce = 0%, Cumulative CPU 679.85 sec
2013-09-18 00:54:25,930 Stage-1 map = 60%, reduce = 0%, Cumulative CPU 679.85 sec
2013-09-18 00:54:27,189 Stage-1 map = 62%, reduce = 0%, Cumulative CPU 679.85 sec
2013-09-18 00:54:28,194 Stage-1 map = 63%, reduce = 0%, Cumulative CPU 679.85 sec
2013-09-18 00:54:29,201 Stage-1 map = 64%, reduce = 0%, Cumulative CPU 679.85 sec
2013-09-18 00:54:30,206 Stage-1 map = 65%, reduce = 0%, Cumulative CPU 679.85 sec
2013-09-18 00:54:31,220 Stage-1 map = 66%, reduce = 0%, Cumulative CPU 679.85 sec
2013-09-18 00:54:32,227 Stage-1 map = 68%, reduce = 0%, Cumulative CPU 726.21 sec
2013-09-18 00:54:33,241 Stage-1 map = 68%, reduce = 0%, Cumulative CPU 726.21 sec
2013-09-18 00:54:34,246 Stage-1 map = 70%, reduce = 0%, Cumulative CPU 726.21 sec
2013-09-18 00:54:35,255 Stage-1 map = 71%, reduce = 0%, Cumulative CPU 726.21 sec
2013-09-18 00:54:36,261 Stage-1 map = 72%, reduce = 0%, Cumulative CPU 726.21 sec
2013-09-18 00:54:37,266 Stage-1 map = 73%, reduce = 0%, Cumulative CPU 726.21 sec
2013-09-18 00:54:38,275 Stage-1 map = 74%, reduce = 0%, Cumulative CPU 726.21 sec
2013-09-18 00:54:39,280 Stage-1 map = 75%, reduce = 0%, Cumulative CPU 2591.6 sec
2013-09-18 00:54:40,290 Stage-1 map = 76%, reduce = 0%, Cumulative CPU 2591.6 sec
2013-09-18 00:54:41,295 Stage-1 map = 77%, reduce = 0%, Cumulative CPU 2591.6 sec
2013-09-18 00:54:42,301 Stage-1 map = 79%, reduce = 0%, Cumulative CPU 2596.52 sec
2013-09-18 00:54:43,327 Stage-1 map = 80%, reduce = 0%, Cumulative CPU 2606.89 sec
2013-09-18 00:54:44,338 Stage-1 map = 83%, reduce = 0%, Cumulative CPU 2623.82 sec
2013-09-18 00:54:45,343 Stage-1 map = 84%, reduce = 0%, Cumulative CPU 2631.82 sec
2013-09-18 00:54:46,349 Stage-1 map = 85%, reduce = 0%, Cumulative CPU 2631.82 sec
2013-09-18 00:54:47,355 Stage-1 map = 87%, reduce = 0%, Cumulative CPU 2647.97 sec
2013-09-18 00:54:48,360 Stage-1 map = 88%, reduce = 0%, Cumulative CPU 2657.08 sec
2013-09-18 00:54:49,367 Stage-1 map = 88%, reduce = 0%, Cumulative CPU 2679.32 sec
2013-09-18 00:54:50,372 Stage-1 map = 90%, reduce = 0%, Cumulative CPU 2712.4 sec
2013-09-18 00:54:51,377 Stage-1 map = 91%, reduce = 0%, Cumulative CPU 2738.4 sec
2013-09-18 00:54:52,383 Stage-1 map = 92%, reduce = 14%, Cumulative CPU 2779.62 sec
2013-09-18 00:54:53,388 Stage-1 map = 93%, reduce = 14%, Cumulative CPU 2809.67 sec
2013-09-18 00:54:54,399 Stage-1 map = 95%, reduce = 14%, Cumulative CPU 2872.97 sec
2013-09-18 00:54:55,404 Stage-1 map = 96%, reduce = 14%, Cumulative CPU 2923.72 sec
2013-09-18 00:54:56,410 Stage-1 map = 97%, reduce = 14%, Cumulative CPU 2923.72 sec
2013-09-18 00:54:57,414 Stage-1 map = 97%, reduce = 14%, Cumulative CPU 2942.34 sec
2013-09-18 00:54:58,419 Stage-1 map = 98%, reduce = 14%, Cumulative CPU 2962.2 sec
2013-09-18 00:54:59,424 Stage-1 map = 98%, reduce = 14%, Cumulative CPU 2983.52 sec
2013-09-18 00:55:00,429 Stage-1 map = 98%, reduce = 14%, Cumulative CPU 3004.99 sec
2013-09-18 00:55:01,435 Stage-1 map = 98%, reduce = 27%, Cumulative CPU 3004.99 sec
2013-09-18 00:55:02,440 Stage-1 map = 99%, reduce = 27%, Cumulative CPU 3004.99 sec
2013-09-18 00:55:03,445 Stage-1 map = 99%, reduce = 27%, Cumulative CPU 3029.77 sec
2013-09-18 00:55:04,450 Stage-1 map = 99%, reduce = 27%, Cumulative CPU 3029.77 sec
2013-09-18 00:55:05,454 Stage-1 map = 99%, reduce = 27%, Cumulative CPU 3029.77 sec
2013-09-18 00:55:06,459 Stage-1 map = 99%, reduce = 27%, Cumulative CPU 3057.4 sec
2013-09-18 00:55:07,464 Stage-1 map = 99%, reduce = 32%, Cumulative CPU 3057.4 sec
2013-09-18 00:55:08,469 Stage-1 map = 99%, reduce = 32%, Cumulative CPU 3057.4 sec
2013-09-18 00:55:09,474 Stage-1 map = 99%, reduce = 32%, Cumulative CPU 3057.4 sec
2013-09-18 00:55:10,479 Stage-1 map = 100%, reduce = 32%, Cumulative CPU 3089.43 sec
2013-09-18 00:55:11,484 Stage-1 map = 100%, reduce = 32%, Cumulative CPU 3089.43 sec
2013-09-18 00:55:12,489 Stage-1 map = 100%, reduce = 32%, Cumulative CPU 3089.43 sec
2013-09-18 00:55:13,495 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 3093.95 sec
2013-09-18 00:55:14,501 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 3093.95 sec
2013-09-18 00:55:15,506 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 3093.95 sec
MapReduce Total cumulative CPU time: 51 minutes 33 seconds 950 msec
Ended Job = job_201309172235_0131
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 1 Cumulative CPU: 3093.95 sec HDFS Read: 77419054 HDFS Write: 1170 SUCCESS
Total MapReduce CPU Time Spent: 51 minutes 33 seconds 950 msec
OK
Time taken: 133.102 seconds, Fetched: 1 row(s)
hive> quit;
times: 3
query: SELECT sum(ResolutionWidth), sum(ResolutionWidth + 1), sum(ResolutionWidth + 2), sum(ResolutionWidth + 3), sum(ResolutionWidth + 4), sum(ResolutionWidth + 5), sum(ResolutionWidth + 6), sum(ResolutionWidth + 7), sum(ResolutionWidth + 8), sum(ResolutionWidth + 9), sum(ResolutionWidth + 10), sum(ResolutionWidth + 11), sum(ResolutionWidth + 12), sum(ResolutionWidth + 13), sum(ResolutionWidth + 14), sum(ResolutionWidth + 15), sum(ResolutionWidth + 16), sum(ResolutionWidth + 17), sum(ResolutionWidth + 18), sum(ResolutionWidth + 19), sum(ResolutionWidth + 20), sum(ResolutionWidth + 21), sum(ResolutionWidth + 22), sum(ResolutionWidth + 23), sum(ResolutionWidth + 24), sum(ResolutionWidth + 25), sum(ResolutionWidth + 26), sum(ResolutionWidth + 27), sum(ResolutionWidth + 28), sum(ResolutionWidth + 29), sum(ResolutionWidth + 30), sum(ResolutionWidth + 31), sum(ResolutionWidth + 32), sum(ResolutionWidth + 33), sum(ResolutionWidth + 34), sum(ResolutionWidth + 35), sum(ResolutionWidth + 36), sum(ResolutionWidth + 37), sum(ResolutionWidth + 38), sum(ResolutionWidth + 39), sum(ResolutionWidth + 40), sum(ResolutionWidth + 41), sum(ResolutionWidth + 42), sum(ResolutionWidth + 43), sum(ResolutionWidth + 44), sum(ResolutionWidth + 45), sum(ResolutionWidth + 46), sum(ResolutionWidth + 47), sum(ResolutionWidth + 48), sum(ResolutionWidth + 49), sum(ResolutionWidth + 50), sum(ResolutionWidth + 51), sum(ResolutionWidth + 52), sum(ResolutionWidth + 53), sum(ResolutionWidth + 54), sum(ResolutionWidth + 55), sum(ResolutionWidth + 56), sum(ResolutionWidth + 57), sum(ResolutionWidth + 58), sum(ResolutionWidth + 59), sum(ResolutionWidth + 60), sum(ResolutionWidth + 61), sum(ResolutionWidth + 62), sum(ResolutionWidth + 63), sum(ResolutionWidth + 64), sum(ResolutionWidth + 65), sum(ResolutionWidth + 66), sum(ResolutionWidth + 67), sum(ResolutionWidth + 68), sum(ResolutionWidth + 69), sum(ResolutionWidth + 70), sum(ResolutionWidth + 71), sum(ResolutionWidth + 72), sum(ResolutionWidth + 73), sum(ResolutionWidth + 74), sum(ResolutionWidth + 75), sum(ResolutionWidth + 76), sum(ResolutionWidth + 77), sum(ResolutionWidth + 78), sum(ResolutionWidth + 79), sum(ResolutionWidth + 80), sum(ResolutionWidth + 81), sum(ResolutionWidth + 82), sum(ResolutionWidth + 83), sum(ResolutionWidth + 84), sum(ResolutionWidth + 85), sum(ResolutionWidth + 86), sum(ResolutionWidth + 87), sum(ResolutionWidth + 88), sum(ResolutionWidth + 89) FROM hits_100m;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_19854@mturlrep13_201309180055_1882171438.txt
hive> SELECT sum(ResolutionWidth), sum(ResolutionWidth + 1), sum(ResolutionWidth + 2), sum(ResolutionWidth + 3), sum(ResolutionWidth + 4), sum(ResolutionWidth + 5), sum(ResolutionWidth + 6), sum(ResolutionWidth + 7), sum(ResolutionWidth + 8), sum(ResolutionWidth + 9), sum(ResolutionWidth + 10), sum(ResolutionWidth + 11), sum(ResolutionWidth + 12), sum(ResolutionWidth + 13), sum(ResolutionWidth + 14), sum(ResolutionWidth + 15), sum(ResolutionWidth + 16), sum(ResolutionWidth + 17), sum(ResolutionWidth + 18), sum(ResolutionWidth + 19), sum(ResolutionWidth + 20), sum(ResolutionWidth + 21), sum(ResolutionWidth + 22), sum(ResolutionWidth + 23), sum(ResolutionWidth + 24), sum(ResolutionWidth + 25), sum(ResolutionWidth + 26), sum(ResolutionWidth + 27), sum(ResolutionWidth + 28), sum(ResolutionWidth + 29), sum(ResolutionWidth + 30), sum(ResolutionWidth + 31), sum(ResolutionWidth + 32), sum(ResolutionWidth + 33), sum(ResolutionWidth + 34), sum(ResolutionWidth + 35), sum(ResolutionWidth + 36), sum(ResolutionWidth + 37), sum(ResolutionWidth + 38), sum(ResolutionWidth + 39), sum(ResolutionWidth + 40), sum(ResolutionWidth + 41), sum(ResolutionWidth + 42), sum(ResolutionWidth + 43), sum(ResolutionWidth + 44), sum(ResolutionWidth + 45), sum(ResolutionWidth + 46), sum(ResolutionWidth + 47), sum(ResolutionWidth + 48), sum(ResolutionWidth + 49), sum(ResolutionWidth + 50), sum(ResolutionWidth + 51), sum(ResolutionWidth + 52), sum(ResolutionWidth + 53), sum(ResolutionWidth + 54), sum(ResolutionWidth + 55), sum(ResolutionWidth + 56), sum(ResolutionWidth + 57), sum(ResolutionWidth + 58), sum(ResolutionWidth + 59), sum(ResolutionWidth + 60), sum(ResolutionWidth + 61), sum(ResolutionWidth + 62), sum(ResolutionWidth + 63), sum(ResolutionWidth + 64), sum(ResolutionWidth + 65), sum(ResolutionWidth + 66), sum(ResolutionWidth + 67), sum(ResolutionWidth + 68), sum(ResolutionWidth + 69), sum(ResolutionWidth + 70), sum(ResolutionWidth + 71), sum(ResolutionWidth + 72), sum(ResolutionWidth + 73), sum(ResolutionWidth + 74), sum(ResolutionWidth + 75), sum(ResolutionWidth + 76), sum(ResolutionWidth + 77), sum(ResolutionWidth + 78), sum(ResolutionWidth + 79), sum(ResolutionWidth + 80), sum(ResolutionWidth + 81), sum(ResolutionWidth + 82), sum(ResolutionWidth + 83), sum(ResolutionWidth + 84), sum(ResolutionWidth + 85), sum(ResolutionWidth + 86), sum(ResolutionWidth + 87), sum(ResolutionWidth + 88), sum(ResolutionWidth + 89) FROM hits_100m;;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0132
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 1
2013-09-18 00:55:27,376 Stage-1 map = 0%, reduce = 0%
2013-09-18 00:55:51,539 Stage-1 map = 1%, reduce = 0%, Cumulative CPU 28.25 sec
2013-09-18 00:55:52,555 Stage-1 map = 2%, reduce = 0%, Cumulative CPU 28.25 sec
2013-09-18 00:55:53,565 Stage-1 map = 5%, reduce = 0%, Cumulative CPU 28.25 sec
2013-09-18 00:55:54,571 Stage-1 map = 7%, reduce = 0%, Cumulative CPU 28.25 sec
2013-09-18 00:55:55,576 Stage-1 map = 10%, reduce = 0%, Cumulative CPU 28.25 sec
2013-09-18 00:55:56,580 Stage-1 map = 12%, reduce = 0%, Cumulative CPU 28.25 sec
2013-09-18 00:55:57,586 Stage-1 map = 13%, reduce = 0%, Cumulative CPU 28.25 sec
2013-09-18 00:55:58,602 Stage-1 map = 13%, reduce = 0%, Cumulative CPU 28.25 sec
2013-09-18 00:55:59,608 Stage-1 map = 14%, reduce = 0%, Cumulative CPU 28.25 sec
2013-09-18 00:56:00,614 Stage-1 map = 14%, reduce = 0%, Cumulative CPU 28.25 sec
2013-09-18 00:56:01,619 Stage-1 map = 14%, reduce = 0%, Cumulative CPU 28.25 sec
2013-09-18 00:56:02,625 Stage-1 map = 15%, reduce = 0%, Cumulative CPU 28.25 sec
2013-09-18 00:56:03,631 Stage-1 map = 15%, reduce = 0%, Cumulative CPU 28.25 sec
2013-09-18 00:56:04,639 Stage-1 map = 17%, reduce = 0%, Cumulative CPU 28.25 sec
2013-09-18 00:56:05,645 Stage-1 map = 19%, reduce = 0%, Cumulative CPU 28.25 sec
2013-09-18 00:56:06,650 Stage-1 map = 20%, reduce = 0%, Cumulative CPU 28.25 sec
2013-09-18 00:56:07,655 Stage-1 map = 22%, reduce = 0%, Cumulative CPU 28.25 sec
2013-09-18 00:56:08,661 Stage-1 map = 24%, reduce = 0%, Cumulative CPU 28.25 sec
2013-09-18 00:56:09,666 Stage-1 map = 25%, reduce = 0%, Cumulative CPU 28.25 sec
2013-09-18 00:56:10,671 Stage-1 map = 27%, reduce = 0%, Cumulative CPU 28.25 sec
2013-09-18 00:56:11,676 Stage-1 map = 28%, reduce = 0%, Cumulative CPU 28.25 sec
2013-09-18 00:56:12,681 Stage-1 map = 28%, reduce = 0%, Cumulative CPU 28.25 sec
2013-09-18 00:56:13,687 Stage-1 map = 29%, reduce = 0%, Cumulative CPU 28.25 sec
2013-09-18 00:56:14,692 Stage-1 map = 29%, reduce = 0%, Cumulative CPU 28.25 sec
2013-09-18 00:56:15,697 Stage-1 map = 32%, reduce = 0%, Cumulative CPU 28.25 sec
2013-09-18 00:56:16,703 Stage-1 map = 32%, reduce = 0%, Cumulative CPU 28.25 sec
2013-09-18 00:56:17,708 Stage-1 map = 34%, reduce = 0%, Cumulative CPU 28.25 sec
2013-09-18 00:56:18,797 Stage-1 map = 34%, reduce = 0%, Cumulative CPU 28.25 sec
2013-09-18 00:56:19,802 Stage-1 map = 35%, reduce = 0%, Cumulative CPU 28.25 sec
2013-09-18 00:56:20,808 Stage-1 map = 36%, reduce = 0%, Cumulative CPU 28.25 sec
2013-09-18 00:56:21,813 Stage-1 map = 37%, reduce = 0%, Cumulative CPU 28.25 sec
2013-09-18 00:56:22,818 Stage-1 map = 39%, reduce = 0%, Cumulative CPU 28.25 sec
2013-09-18 00:56:23,824 Stage-1 map = 40%, reduce = 0%, Cumulative CPU 28.25 sec
2013-09-18 00:56:24,829 Stage-1 map = 42%, reduce = 0%, Cumulative CPU 28.25 sec
2013-09-18 00:56:25,834 Stage-1 map = 43%, reduce = 0%, Cumulative CPU 28.25 sec
2013-09-18 00:56:26,840 Stage-1 map = 44%, reduce = 0%, Cumulative CPU 28.25 sec
2013-09-18 00:56:27,845 Stage-1 map = 46%, reduce = 0%, Cumulative CPU 28.25 sec
2013-09-18 00:56:28,850 Stage-1 map = 46%, reduce = 0%, Cumulative CPU 28.25 sec
2013-09-18 00:56:29,855 Stage-1 map = 47%, reduce = 0%, Cumulative CPU 28.25 sec
2013-09-18 00:56:30,861 Stage-1 map = 48%, reduce = 0%, Cumulative CPU 28.25 sec
2013-09-18 00:56:31,866 Stage-1 map = 48%, reduce = 0%, Cumulative CPU 28.25 sec
2013-09-18 00:56:32,871 Stage-1 map = 50%, reduce = 0%, Cumulative CPU 28.25 sec
2013-09-18 00:56:33,876 Stage-1 map = 51%, reduce = 0%, Cumulative CPU 28.25 sec
2013-09-18 00:56:34,881 Stage-1 map = 53%, reduce = 0%, Cumulative CPU 28.25 sec
2013-09-18 00:56:35,887 Stage-1 map = 54%, reduce = 0%, Cumulative CPU 28.25 sec
2013-09-18 00:56:36,899 Stage-1 map = 56%, reduce = 0%, Cumulative CPU 28.25 sec
2013-09-18 00:56:38,073 Stage-1 map = 56%, reduce = 0%, Cumulative CPU 28.25 sec
2013-09-18 00:56:39,078 Stage-1 map = 58%, reduce = 0%, Cumulative CPU 28.25 sec
2013-09-18 00:56:40,093 Stage-1 map = 60%, reduce = 0%, Cumulative CPU 2125.98 sec
2013-09-18 00:56:41,098 Stage-1 map = 60%, reduce = 0%, Cumulative CPU 2125.98 sec
2013-09-18 00:56:42,104 Stage-1 map = 61%, reduce = 0%, Cumulative CPU 2125.98 sec
2013-09-18 00:56:43,109 Stage-1 map = 63%, reduce = 0%, Cumulative CPU 2125.98 sec
2013-09-18 00:56:44,116 Stage-1 map = 64%, reduce = 0%, Cumulative CPU 2129.75 sec
2013-09-18 00:56:45,121 Stage-1 map = 65%, reduce = 0%, Cumulative CPU 2129.75 sec
2013-09-18 00:56:46,127 Stage-1 map = 66%, reduce = 0%, Cumulative CPU 2129.75 sec
2013-09-18 00:56:47,136 Stage-1 map = 68%, reduce = 0%, Cumulative CPU 2129.75 sec
2013-09-18 00:56:48,141 Stage-1 map = 69%, reduce = 0%, Cumulative CPU 2129.75 sec
2013-09-18 00:56:49,147 Stage-1 map = 71%, reduce = 0%, Cumulative CPU 2129.75 sec
2013-09-18 00:56:50,152 Stage-1 map = 72%, reduce = 0%, Cumulative CPU 2129.75 sec
2013-09-18 00:56:51,158 Stage-1 map = 72%, reduce = 0%, Cumulative CPU 2129.75 sec
2013-09-18 00:56:52,163 Stage-1 map = 73%, reduce = 0%, Cumulative CPU 2129.75 sec
2013-09-18 00:56:53,168 Stage-1 map = 75%, reduce = 0%, Cumulative CPU 2129.75 sec
2013-09-18 00:56:54,174 Stage-1 map = 76%, reduce = 0%, Cumulative CPU 2129.75 sec
2013-09-18 00:56:55,179 Stage-1 map = 76%, reduce = 0%, Cumulative CPU 2144.3 sec
2013-09-18 00:56:56,184 Stage-1 map = 77%, reduce = 0%, Cumulative CPU 2144.3 sec
2013-09-18 00:56:57,189 Stage-1 map = 78%, reduce = 0%, Cumulative CPU 2160.94 sec
2013-09-18 00:56:58,195 Stage-1 map = 80%, reduce = 0%, Cumulative CPU 2178.2 sec
2013-09-18 00:56:59,200 Stage-1 map = 81%, reduce = 0%, Cumulative CPU 2178.2 sec
2013-09-18 00:57:00,205 Stage-1 map = 83%, reduce = 0%, Cumulative CPU 2199.66 sec
2013-09-18 00:57:01,210 Stage-1 map = 84%, reduce = 0%, Cumulative CPU 2218.05 sec
2013-09-18 00:57:02,216 Stage-1 map = 86%, reduce = 0%, Cumulative CPU 2282.19 sec
2013-09-18 00:57:03,221 Stage-1 map = 87%, reduce = 0%, Cumulative CPU 2304.47 sec
2013-09-18 00:57:04,227 Stage-1 map = 88%, reduce = 0%, Cumulative CPU 2349.8 sec
2013-09-18 00:57:05,233 Stage-1 map = 89%, reduce = 0%, Cumulative CPU 2372.02 sec
2013-09-18 00:57:06,242 Stage-1 map = 90%, reduce = 0%, Cumulative CPU 2427.89 sec
2013-09-18 00:57:07,248 Stage-1 map = 91%, reduce = 0%, Cumulative CPU 2503.41 sec
2013-09-18 00:57:08,254 Stage-1 map = 94%, reduce = 11%, Cumulative CPU 2613.26 sec
2013-09-18 00:57:09,259 Stage-1 map = 95%, reduce = 11%, Cumulative CPU 2701.51 sec
2013-09-18 00:57:10,265 Stage-1 map = 96%, reduce = 11%, Cumulative CPU 2758.3 sec
2013-09-18 00:57:11,271 Stage-1 map = 97%, reduce = 22%, Cumulative CPU 2875.86 sec
2013-09-18 00:57:12,278 Stage-1 map = 97%, reduce = 22%, Cumulative CPU 2875.86 sec
2013-09-18 00:57:13,284 Stage-1 map = 98%, reduce = 22%, Cumulative CPU 2941.83 sec
2013-09-18 00:57:14,290 Stage-1 map = 99%, reduce = 22%, Cumulative CPU 3011.19 sec
2013-09-18 00:57:15,296 Stage-1 map = 99%, reduce = 22%, Cumulative CPU 3011.19 sec
2013-09-18 00:57:16,302 Stage-1 map = 99%, reduce = 22%, Cumulative CPU 3011.19 sec
2013-09-18 00:57:17,307 Stage-1 map = 99%, reduce = 22%, Cumulative CPU 3011.19 sec
2013-09-18 00:57:18,313 Stage-1 map = 99%, reduce = 22%, Cumulative CPU 3011.19 sec
2013-09-18 00:57:19,319 Stage-1 map = 99%, reduce = 22%, Cumulative CPU 3011.19 sec
2013-09-18 00:57:20,325 Stage-1 map = 99%, reduce = 32%, Cumulative CPU 3011.19 sec
2013-09-18 00:57:21,330 Stage-1 map = 99%, reduce = 32%, Cumulative CPU 3011.19 sec
2013-09-18 00:57:22,335 Stage-1 map = 99%, reduce = 32%, Cumulative CPU 3011.19 sec
2013-09-18 00:57:23,340 Stage-1 map = 99%, reduce = 32%, Cumulative CPU 3053.74 sec
2013-09-18 00:57:24,344 Stage-1 map = 99%, reduce = 32%, Cumulative CPU 3053.74 sec
2013-09-18 00:57:25,348 Stage-1 map = 100%, reduce = 32%, Cumulative CPU 3099.46 sec
2013-09-18 00:57:26,352 Stage-1 map = 100%, reduce = 32%, Cumulative CPU 3099.46 sec
2013-09-18 00:57:27,356 Stage-1 map = 100%, reduce = 32%, Cumulative CPU 3099.46 sec
2013-09-18 00:57:28,359 Stage-1 map = 100%, reduce = 32%, Cumulative CPU 3099.46 sec
2013-09-18 00:57:29,364 Stage-1 map = 100%, reduce = 32%, Cumulative CPU 3099.46 sec
2013-09-18 00:57:30,368 Stage-1 map = 100%, reduce = 32%, Cumulative CPU 3099.46 sec
2013-09-18 00:57:31,373 Stage-1 map = 100%, reduce = 32%, Cumulative CPU 3099.46 sec
2013-09-18 00:57:32,380 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 3104.28 sec
2013-09-18 00:57:33,385 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 3104.28 sec
MapReduce Total cumulative CPU time: 51 minutes 44 seconds 280 msec
Ended Job = job_201309172235_0132
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 1 Cumulative CPU: 3104.28 sec HDFS Read: 77419054 HDFS Write: 1170 SUCCESS
Total MapReduce CPU Time Spent: 51 minutes 44 seconds 280 msec
OK
Time taken: 135.896 seconds, Fetched: 1 row(s)
hive> quit;
-- много тупых агрегатных функций.;
times: 1
query: SELECT SearchEngineID, ClientIP, count(*) AS c, sum(Refresh), avg(ResolutionWidth) FROM hits_100m WHERE SearchPhrase != '' GROUP BY SearchEngineID, ClientIP ORDER BY c DESC LIMIT 10;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_25111@mturlrep13_201309180057_1190943875.txt
hive> SELECT SearchEngineID, ClientIP, count(*) AS c, sum(Refresh), avg(ResolutionWidth) FROM hits_100m WHERE SearchPhrase != '' GROUP BY SearchEngineID, ClientIP ORDER BY c DESC LIMIT 10;;
Total MapReduce jobs = 2
Launching Job 1 out of 2
Number of reduce tasks not specified. Estimated from input data size: 11
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0133
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 11
2013-09-18 00:57:50,216 Stage-1 map = 0%, reduce = 0%
2013-09-18 00:58:00,383 Stage-1 map = 2%, reduce = 0%
2013-09-18 00:58:04,415 Stage-1 map = 8%, reduce = 0%
2013-09-18 00:58:07,430 Stage-1 map = 24%, reduce = 0%
2013-09-18 00:58:10,044 Stage-1 map = 33%, reduce = 0%
2013-09-18 00:58:11,136 Stage-1 map = 37%, reduce = 0%
2013-09-18 00:58:13,987 Stage-1 map = 45%, reduce = 0%, Cumulative CPU 10.4 sec
2013-09-18 00:58:14,998 Stage-1 map = 47%, reduce = 0%, Cumulative CPU 10.4 sec
2013-09-18 00:58:16,016 Stage-1 map = 48%, reduce = 0%, Cumulative CPU 44.12 sec
2013-09-18 00:58:17,029 Stage-1 map = 60%, reduce = 0%, Cumulative CPU 44.12 sec
2013-09-18 00:58:19,269 Stage-1 map = 61%, reduce = 0%, Cumulative CPU 44.12 sec
2013-09-18 00:58:20,277 Stage-1 map = 71%, reduce = 0%, Cumulative CPU 54.05 sec
2013-09-18 00:58:21,283 Stage-1 map = 71%, reduce = 0%, Cumulative CPU 54.05 sec
2013-09-18 00:58:22,290 Stage-1 map = 81%, reduce = 0%, Cumulative CPU 217.63 sec
2013-09-18 00:58:23,297 Stage-1 map = 89%, reduce = 0%, Cumulative CPU 293.68 sec
2013-09-18 00:58:24,303 Stage-1 map = 91%, reduce = 0%, Cumulative CPU 354.53 sec
2013-09-18 00:58:25,310 Stage-1 map = 97%, reduce = 0%, Cumulative CPU 431.46 sec
2013-09-18 00:58:26,316 Stage-1 map = 100%, reduce = 6%, Cumulative CPU 532.5 sec
2013-09-18 00:58:27,322 Stage-1 map = 100%, reduce = 9%, Cumulative CPU 532.5 sec
2013-09-18 00:58:28,329 Stage-1 map = 100%, reduce = 9%, Cumulative CPU 532.5 sec
2013-09-18 00:58:29,337 Stage-1 map = 100%, reduce = 27%, Cumulative CPU 532.5 sec
2013-09-18 00:58:30,344 Stage-1 map = 100%, reduce = 27%, Cumulative CPU 532.5 sec
2013-09-18 00:58:31,351 Stage-1 map = 100%, reduce = 27%, Cumulative CPU 532.5 sec
2013-09-18 00:58:32,359 Stage-1 map = 100%, reduce = 52%, Cumulative CPU 532.5 sec
2013-09-18 00:58:33,367 Stage-1 map = 100%, reduce = 52%, Cumulative CPU 532.5 sec
2013-09-18 00:58:34,374 Stage-1 map = 100%, reduce = 52%, Cumulative CPU 532.5 sec
2013-09-18 00:58:35,379 Stage-1 map = 100%, reduce = 79%, Cumulative CPU 532.5 sec
2013-09-18 00:58:36,387 Stage-1 map = 100%, reduce = 88%, Cumulative CPU 600.15 sec
2013-09-18 00:58:37,394 Stage-1 map = 100%, reduce = 88%, Cumulative CPU 600.15 sec
2013-09-18 00:58:38,402 Stage-1 map = 100%, reduce = 98%, Cumulative CPU 622.76 sec
2013-09-18 00:58:39,408 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 656.89 sec
2013-09-18 00:58:40,415 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 656.89 sec
MapReduce Total cumulative CPU time: 10 minutes 56 seconds 890 msec
Ended Job = job_201309172235_0133
Launching Job 2 out of 2
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0134
Hadoop job information for Stage-2: number of mappers: 1; number of reducers: 1
2013-09-18 00:58:43,889 Stage-2 map = 0%, reduce = 0%
2013-09-18 00:58:50,910 Stage-2 map = 9%, reduce = 0%
2013-09-18 00:58:53,919 Stage-2 map = 18%, reduce = 0%
2013-09-18 00:58:56,928 Stage-2 map = 27%, reduce = 0%
2013-09-18 00:58:59,938 Stage-2 map = 45%, reduce = 0%
2013-09-18 00:59:02,948 Stage-2 map = 55%, reduce = 0%
2013-09-18 00:59:05,958 Stage-2 map = 64%, reduce = 0%
2013-09-18 00:59:08,967 Stage-2 map = 73%, reduce = 0%
2013-09-18 00:59:11,976 Stage-2 map = 82%, reduce = 0%
2013-09-18 00:59:14,991 Stage-2 map = 91%, reduce = 0%
2013-09-18 00:59:20,012 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 43.91 sec
2013-09-18 00:59:21,017 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 43.91 sec
2013-09-18 00:59:22,021 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 43.91 sec
2013-09-18 00:59:23,025 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 43.91 sec
2013-09-18 00:59:24,029 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 43.91 sec
2013-09-18 00:59:25,033 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 43.91 sec
2013-09-18 00:59:26,038 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 43.91 sec
2013-09-18 00:59:27,043 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 43.91 sec
2013-09-18 00:59:28,047 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 43.91 sec
2013-09-18 00:59:29,052 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 43.91 sec
2013-09-18 00:59:30,056 Stage-2 map = 100%, reduce = 68%, Cumulative CPU 43.91 sec
2013-09-18 00:59:31,065 Stage-2 map = 100%, reduce = 68%, Cumulative CPU 43.91 sec
2013-09-18 00:59:32,070 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 50.31 sec
2013-09-18 00:59:33,075 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 50.31 sec
MapReduce Total cumulative CPU time: 50 seconds 310 msec
Ended Job = job_201309172235_0134
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 11 Cumulative CPU: 656.89 sec HDFS Read: 613013229 HDFS Write: 191007594 SUCCESS
Job 1: Map: 1 Reduce: 1 Cumulative CPU: 50.31 sec HDFS Read: 191011171 HDFS Write: 399 SUCCESS
Total MapReduce CPU Time Spent: 11 minutes 47 seconds 200 msec
OK
Time taken: 113.348 seconds, Fetched: 10 row(s)
hive> quit;
times: 2
query: SELECT SearchEngineID, ClientIP, count(*) AS c, sum(Refresh), avg(ResolutionWidth) FROM hits_100m WHERE SearchPhrase != '' GROUP BY SearchEngineID, ClientIP ORDER BY c DESC LIMIT 10;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_32655@mturlrep13_201309180059_1219302235.txt
hive> SELECT SearchEngineID, ClientIP, count(*) AS c, sum(Refresh), avg(ResolutionWidth) FROM hits_100m WHERE SearchPhrase != '' GROUP BY SearchEngineID, ClientIP ORDER BY c DESC LIMIT 10;;
Total MapReduce jobs = 2
Launching Job 1 out of 2
Number of reduce tasks not specified. Estimated from input data size: 11
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0135
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 11
2013-09-18 00:59:42,709 Stage-1 map = 0%, reduce = 0%
2013-09-18 00:59:54,021 Stage-1 map = 2%, reduce = 0%
2013-09-18 00:59:55,026 Stage-1 map = 6%, reduce = 0%
2013-09-18 00:59:56,031 Stage-1 map = 13%, reduce = 0%
2013-09-18 00:59:57,047 Stage-1 map = 19%, reduce = 0%
2013-09-18 00:59:58,067 Stage-1 map = 26%, reduce = 0%
2013-09-18 00:59:59,081 Stage-1 map = 39%, reduce = 0%
2013-09-18 01:00:00,095 Stage-1 map = 44%, reduce = 0%, Cumulative CPU 11.06 sec
2013-09-18 01:00:01,103 Stage-1 map = 50%, reduce = 0%, Cumulative CPU 22.66 sec
2013-09-18 01:00:02,114 Stage-1 map = 70%, reduce = 0%, Cumulative CPU 93.97 sec
2013-09-18 01:00:03,127 Stage-1 map = 74%, reduce = 0%, Cumulative CPU 133.19 sec
2013-09-18 01:00:04,152 Stage-1 map = 81%, reduce = 0%, Cumulative CPU 186.98 sec
2013-09-18 01:00:05,168 Stage-1 map = 89%, reduce = 0%, Cumulative CPU 248.68 sec
2013-09-18 01:00:06,178 Stage-1 map = 94%, reduce = 0%, Cumulative CPU 281.8 sec
2013-09-18 01:00:07,185 Stage-1 map = 99%, reduce = 0%, Cumulative CPU 545.59 sec
2013-09-18 01:00:08,191 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 563.72 sec
2013-09-18 01:00:09,197 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 563.72 sec
2013-09-18 01:00:10,203 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 563.72 sec
2013-09-18 01:00:11,210 Stage-1 map = 100%, reduce = 9%, Cumulative CPU 563.72 sec
2013-09-18 01:00:12,218 Stage-1 map = 100%, reduce = 21%, Cumulative CPU 563.72 sec
2013-09-18 01:00:13,233 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 563.72 sec
2013-09-18 01:00:14,240 Stage-1 map = 100%, reduce = 39%, Cumulative CPU 563.72 sec
2013-09-18 01:00:15,247 Stage-1 map = 100%, reduce = 55%, Cumulative CPU 563.72 sec
2013-09-18 01:00:16,253 Stage-1 map = 100%, reduce = 67%, Cumulative CPU 563.72 sec
2013-09-18 01:00:17,261 Stage-1 map = 100%, reduce = 71%, Cumulative CPU 563.72 sec
2013-09-18 01:00:18,270 Stage-1 map = 100%, reduce = 82%, Cumulative CPU 574.8 sec
2013-09-18 01:00:19,278 Stage-1 map = 100%, reduce = 94%, Cumulative CPU 609.56 sec
2013-09-18 01:00:20,285 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 690.28 sec
2013-09-18 01:00:21,291 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 690.28 sec
2013-09-18 01:00:22,299 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 690.28 sec
MapReduce Total cumulative CPU time: 11 minutes 30 seconds 280 msec
Ended Job = job_201309172235_0135
Launching Job 2 out of 2
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0136
Hadoop job information for Stage-2: number of mappers: 1; number of reducers: 1
2013-09-18 01:00:25,765 Stage-2 map = 0%, reduce = 0%
2013-09-18 01:00:32,789 Stage-2 map = 9%, reduce = 0%
2013-09-18 01:00:35,800 Stage-2 map = 18%, reduce = 0%
2013-09-18 01:00:38,811 Stage-2 map = 27%, reduce = 0%
2013-09-18 01:00:41,823 Stage-2 map = 45%, reduce = 0%, Cumulative CPU 17.28 sec
2013-09-18 01:00:42,828 Stage-2 map = 45%, reduce = 0%, Cumulative CPU 17.28 sec
2013-09-18 01:00:43,834 Stage-2 map = 45%, reduce = 0%, Cumulative CPU 17.28 sec
2013-09-18 01:00:44,839 Stage-2 map = 55%, reduce = 0%, Cumulative CPU 17.28 sec
2013-09-18 01:00:45,844 Stage-2 map = 55%, reduce = 0%, Cumulative CPU 17.28 sec
2013-09-18 01:00:46,848 Stage-2 map = 55%, reduce = 0%, Cumulative CPU 17.28 sec
2013-09-18 01:00:47,853 Stage-2 map = 64%, reduce = 0%, Cumulative CPU 17.28 sec
2013-09-18 01:00:48,859 Stage-2 map = 64%, reduce = 0%, Cumulative CPU 17.28 sec
2013-09-18 01:00:49,864 Stage-2 map = 64%, reduce = 0%, Cumulative CPU 17.28 sec
2013-09-18 01:00:50,869 Stage-2 map = 73%, reduce = 0%, Cumulative CPU 17.28 sec
2013-09-18 01:00:51,874 Stage-2 map = 73%, reduce = 0%, Cumulative CPU 17.28 sec
2013-09-18 01:00:52,879 Stage-2 map = 73%, reduce = 0%, Cumulative CPU 17.28 sec
2013-09-18 01:00:53,885 Stage-2 map = 91%, reduce = 0%, Cumulative CPU 17.28 sec
2013-09-18 01:00:54,890 Stage-2 map = 91%, reduce = 0%, Cumulative CPU 17.28 sec
2013-09-18 01:00:55,895 Stage-2 map = 91%, reduce = 0%, Cumulative CPU 17.28 sec
2013-09-18 01:00:56,901 Stage-2 map = 91%, reduce = 0%, Cumulative CPU 17.28 sec
2013-09-18 01:00:57,906 Stage-2 map = 91%, reduce = 0%, Cumulative CPU 17.28 sec
2013-09-18 01:00:58,911 Stage-2 map = 91%, reduce = 0%, Cumulative CPU 17.28 sec
2013-09-18 01:00:59,916 Stage-2 map = 91%, reduce = 0%, Cumulative CPU 17.28 sec
2013-09-18 01:01:00,921 Stage-2 map = 91%, reduce = 0%, Cumulative CPU 17.28 sec
2013-09-18 01:01:01,926 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 43.57 sec
2013-09-18 01:01:02,936 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 43.57 sec
2013-09-18 01:01:03,941 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 43.57 sec
2013-09-18 01:01:04,945 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 43.57 sec
2013-09-18 01:01:05,949 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 43.57 sec
2013-09-18 01:01:06,954 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 43.57 sec
2013-09-18 01:01:07,958 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 43.57 sec
2013-09-18 01:01:08,963 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 43.57 sec
2013-09-18 01:01:09,968 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 43.57 sec
2013-09-18 01:01:10,972 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 43.57 sec
2013-09-18 01:01:11,976 Stage-2 map = 100%, reduce = 70%, Cumulative CPU 43.57 sec
2013-09-18 01:01:12,981 Stage-2 map = 100%, reduce = 70%, Cumulative CPU 43.57 sec
2013-09-18 01:01:13,986 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 49.9 sec
2013-09-18 01:01:14,991 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 49.9 sec
MapReduce Total cumulative CPU time: 49 seconds 900 msec
Ended Job = job_201309172235_0136
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 11 Cumulative CPU: 690.28 sec HDFS Read: 613013229 HDFS Write: 191007594 SUCCESS
Job 1: Map: 1 Reduce: 1 Cumulative CPU: 49.9 sec HDFS Read: 191011171 HDFS Write: 399 SUCCESS
Total MapReduce CPU Time Spent: 12 minutes 20 seconds 180 msec
OK
Time taken: 100.032 seconds, Fetched: 10 row(s)
hive> quit;
times: 3
query: SELECT SearchEngineID, ClientIP, count(*) AS c, sum(Refresh), avg(ResolutionWidth) FROM hits_100m WHERE SearchPhrase != '' GROUP BY SearchEngineID, ClientIP ORDER BY c DESC LIMIT 10;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_8933@mturlrep13_201309180101_1921637373.txt
hive> SELECT SearchEngineID, ClientIP, count(*) AS c, sum(Refresh), avg(ResolutionWidth) FROM hits_100m WHERE SearchPhrase != '' GROUP BY SearchEngineID, ClientIP ORDER BY c DESC LIMIT 10;;
Total MapReduce jobs = 2
Launching Job 1 out of 2
Number of reduce tasks not specified. Estimated from input data size: 11
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0137
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 11
2013-09-18 01:01:24,685 Stage-1 map = 0%, reduce = 0%
2013-09-18 01:01:35,760 Stage-1 map = 2%, reduce = 0%
2013-09-18 01:01:36,769 Stage-1 map = 7%, reduce = 0%
2013-09-18 01:01:37,774 Stage-1 map = 12%, reduce = 0%
2013-09-18 01:01:38,803 Stage-1 map = 17%, reduce = 0%
2013-09-18 01:01:39,847 Stage-1 map = 23%, reduce = 0%
2013-09-18 01:01:40,868 Stage-1 map = 39%, reduce = 0%, Cumulative CPU 286.36 sec
2013-09-18 01:01:41,883 Stage-1 map = 44%, reduce = 0%, Cumulative CPU 286.36 sec
2013-09-18 01:01:42,895 Stage-1 map = 48%, reduce = 0%, Cumulative CPU 289.4 sec
2013-09-18 01:01:43,902 Stage-1 map = 66%, reduce = 0%, Cumulative CPU 301.74 sec
2013-09-18 01:01:44,920 Stage-1 map = 74%, reduce = 0%, Cumulative CPU 341.22 sec
2013-09-18 01:01:45,934 Stage-1 map = 80%, reduce = 0%, Cumulative CPU 365.65 sec
2013-09-18 01:01:46,942 Stage-1 map = 88%, reduce = 0%, Cumulative CPU 373.98 sec
2013-09-18 01:01:47,949 Stage-1 map = 96%, reduce = 0%, Cumulative CPU 460.32 sec
2013-09-18 01:01:48,957 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 557.15 sec
2013-09-18 01:01:49,963 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 557.15 sec
2013-09-18 01:01:50,969 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 557.15 sec
2013-09-18 01:01:51,974 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 557.15 sec
2013-09-18 01:01:52,986 Stage-1 map = 100%, reduce = 3%, Cumulative CPU 557.15 sec
2013-09-18 01:01:53,993 Stage-1 map = 100%, reduce = 18%, Cumulative CPU 557.15 sec
2013-09-18 01:01:55,000 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 557.15 sec
2013-09-18 01:01:56,006 Stage-1 map = 100%, reduce = 36%, Cumulative CPU 557.15 sec
2013-09-18 01:01:57,013 Stage-1 map = 100%, reduce = 52%, Cumulative CPU 557.15 sec
2013-09-18 01:01:58,020 Stage-1 map = 100%, reduce = 67%, Cumulative CPU 557.15 sec
2013-09-18 01:01:59,026 Stage-1 map = 100%, reduce = 69%, Cumulative CPU 557.15 sec
2013-09-18 01:02:00,035 Stage-1 map = 100%, reduce = 80%, Cumulative CPU 568.49 sec
2013-09-18 01:02:01,042 Stage-1 map = 100%, reduce = 95%, Cumulative CPU 603.17 sec
2013-09-18 01:02:02,048 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 671.6 sec
2013-09-18 01:02:03,056 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 682.9 sec
2013-09-18 01:02:04,062 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 682.9 sec
MapReduce Total cumulative CPU time: 11 minutes 22 seconds 900 msec
Ended Job = job_201309172235_0137
Launching Job 2 out of 2
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0138
Hadoop job information for Stage-2: number of mappers: 1; number of reducers: 1
2013-09-18 01:02:07,547 Stage-2 map = 0%, reduce = 0%
2013-09-18 01:02:14,570 Stage-2 map = 9%, reduce = 0%
2013-09-18 01:02:17,581 Stage-2 map = 18%, reduce = 0%
2013-09-18 01:02:20,591 Stage-2 map = 27%, reduce = 0%
2013-09-18 01:02:23,602 Stage-2 map = 45%, reduce = 0%
2013-09-18 01:02:26,613 Stage-2 map = 55%, reduce = 0%
2013-09-18 01:02:29,623 Stage-2 map = 64%, reduce = 0%
2013-09-18 01:02:32,634 Stage-2 map = 73%, reduce = 0%
2013-09-18 01:02:35,643 Stage-2 map = 91%, reduce = 0%
2013-09-18 01:02:43,669 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 42.8 sec
2013-09-18 01:02:44,674 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 42.8 sec
2013-09-18 01:02:45,679 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 42.8 sec
2013-09-18 01:02:46,684 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 42.8 sec
2013-09-18 01:02:47,688 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 42.8 sec
2013-09-18 01:02:48,693 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 42.8 sec
2013-09-18 01:02:49,698 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 42.8 sec
2013-09-18 01:02:50,704 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 42.8 sec
2013-09-18 01:02:51,708 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 42.8 sec
2013-09-18 01:02:52,713 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 42.8 sec
2013-09-18 01:02:53,717 Stage-2 map = 100%, reduce = 70%, Cumulative CPU 42.8 sec
2013-09-18 01:02:54,722 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 49.05 sec
2013-09-18 01:02:55,728 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 49.05 sec
2013-09-18 01:02:56,733 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 49.05 sec
MapReduce Total cumulative CPU time: 49 seconds 50 msec
Ended Job = job_201309172235_0138
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 11 Cumulative CPU: 682.9 sec HDFS Read: 613013229 HDFS Write: 191007594 SUCCESS
Job 1: Map: 1 Reduce: 1 Cumulative CPU: 49.05 sec HDFS Read: 191011171 HDFS Write: 399 SUCCESS
Total MapReduce CPU Time Spent: 12 minutes 11 seconds 950 msec
OK
Time taken: 99.905 seconds, Fetched: 10 row(s)
hive> quit;
-- сложная агрегация, для больших таблиц может не хватить оперативки.;
times: 1
query: SELECT WatchID, ClientIP, count(*) AS c, sum(Refresh), avg(ResolutionWidth) FROM hits_100m WHERE SearchPhrase != '' GROUP BY WatchID, ClientIP ORDER BY c DESC LIMIT 10;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_16458@mturlrep13_201309180103_1014086590.txt
hive> SELECT WatchID, ClientIP, count(*) AS c, sum(Refresh), avg(ResolutionWidth) FROM hits_100m WHERE SearchPhrase != '' GROUP BY WatchID, ClientIP ORDER BY c DESC LIMIT 10;;
Total MapReduce jobs = 2
Launching Job 1 out of 2
Number of reduce tasks not specified. Estimated from input data size: 11
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0139
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 11
2013-09-18 01:03:15,821 Stage-1 map = 0%, reduce = 0%
2013-09-18 01:03:25,881 Stage-1 map = 2%, reduce = 0%
2013-09-18 01:03:28,896 Stage-1 map = 3%, reduce = 0%
2013-09-18 01:03:30,685 Stage-1 map = 11%, reduce = 0%
2013-09-18 01:03:31,692 Stage-1 map = 12%, reduce = 0%
2013-09-18 01:03:33,713 Stage-1 map = 24%, reduce = 0%
2013-09-18 01:03:34,719 Stage-1 map = 25%, reduce = 0%
2013-09-18 01:03:41,834 Stage-1 map = 52%, reduce = 0%, Cumulative CPU 361.72 sec
2013-09-18 01:03:42,856 Stage-1 map = 61%, reduce = 0%, Cumulative CPU 361.72 sec
2013-09-18 01:03:43,882 Stage-1 map = 65%, reduce = 0%, Cumulative CPU 363.16 sec
2013-09-18 01:03:44,896 Stage-1 map = 69%, reduce = 0%, Cumulative CPU 378.03 sec
2013-09-18 01:03:45,904 Stage-1 map = 78%, reduce = 0%, Cumulative CPU 403.06 sec
2013-09-18 01:03:46,911 Stage-1 map = 81%, reduce = 0%, Cumulative CPU 418.77 sec
2013-09-18 01:03:48,058 Stage-1 map = 84%, reduce = 0%, Cumulative CPU 434.02 sec
2013-09-18 01:03:49,066 Stage-1 map = 90%, reduce = 0%, Cumulative CPU 460.07 sec
2013-09-18 01:03:50,076 Stage-1 map = 97%, reduce = 0%, Cumulative CPU 509.72 sec
2013-09-18 01:03:51,083 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 547.13 sec
2013-09-18 01:03:52,089 Stage-1 map = 100%, reduce = 2%, Cumulative CPU 547.13 sec
2013-09-18 01:03:53,096 Stage-1 map = 100%, reduce = 8%, Cumulative CPU 547.13 sec
2013-09-18 01:03:54,105 Stage-1 map = 100%, reduce = 20%, Cumulative CPU 547.13 sec
2013-09-18 01:03:55,112 Stage-1 map = 100%, reduce = 36%, Cumulative CPU 547.13 sec
2013-09-18 01:03:56,118 Stage-1 map = 100%, reduce = 39%, Cumulative CPU 547.13 sec
2013-09-18 01:03:57,125 Stage-1 map = 100%, reduce = 52%, Cumulative CPU 547.13 sec
2013-09-18 01:03:58,132 Stage-1 map = 100%, reduce = 65%, Cumulative CPU 547.13 sec
2013-09-18 01:03:59,138 Stage-1 map = 100%, reduce = 67%, Cumulative CPU 547.13 sec
2013-09-18 01:04:00,145 Stage-1 map = 100%, reduce = 72%, Cumulative CPU 547.13 sec
2013-09-18 01:04:01,154 Stage-1 map = 100%, reduce = 78%, Cumulative CPU 560.31 sec
2013-09-18 01:04:02,162 Stage-1 map = 100%, reduce = 85%, Cumulative CPU 573.82 sec
2013-09-18 01:04:03,169 Stage-1 map = 100%, reduce = 87%, Cumulative CPU 600.97 sec
2013-09-18 01:04:04,176 Stage-1 map = 100%, reduce = 98%, Cumulative CPU 683.89 sec
2013-09-18 01:04:05,182 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 683.89 sec
2013-09-18 01:04:06,188 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 697.41 sec
2013-09-18 01:04:07,193 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 697.41 sec
MapReduce Total cumulative CPU time: 11 minutes 37 seconds 410 msec
Ended Job = job_201309172235_0139
Launching Job 2 out of 2
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0140
Hadoop job information for Stage-2: number of mappers: 2; number of reducers: 1
2013-09-18 01:04:10,746 Stage-2 map = 0%, reduce = 0%
2013-09-18 01:04:20,776 Stage-2 map = 10%, reduce = 0%
2013-09-18 01:04:23,786 Stage-2 map = 18%, reduce = 0%
2013-09-18 01:04:26,796 Stage-2 map = 28%, reduce = 0%
2013-09-18 01:04:29,806 Stage-2 map = 37%, reduce = 0%
2013-09-18 01:04:32,817 Stage-2 map = 55%, reduce = 0%
2013-09-18 01:04:38,835 Stage-2 map = 73%, reduce = 0%
2013-09-18 01:04:44,858 Stage-2 map = 82%, reduce = 0%, Cumulative CPU 69.67 sec
2013-09-18 01:04:45,863 Stage-2 map = 82%, reduce = 0%, Cumulative CPU 69.67 sec
2013-09-18 01:04:46,868 Stage-2 map = 82%, reduce = 0%, Cumulative CPU 69.67 sec
2013-09-18 01:04:47,873 Stage-2 map = 82%, reduce = 0%, Cumulative CPU 69.67 sec
2013-09-18 01:04:48,879 Stage-2 map = 82%, reduce = 0%, Cumulative CPU 69.67 sec
2013-09-18 01:04:49,884 Stage-2 map = 92%, reduce = 0%, Cumulative CPU 81.11 sec
2013-09-18 01:04:50,889 Stage-2 map = 92%, reduce = 0%, Cumulative CPU 81.11 sec
2013-09-18 01:04:51,894 Stage-2 map = 92%, reduce = 0%, Cumulative CPU 81.11 sec
2013-09-18 01:04:52,898 Stage-2 map = 92%, reduce = 0%, Cumulative CPU 81.11 sec
2013-09-18 01:04:53,903 Stage-2 map = 92%, reduce = 0%, Cumulative CPU 81.11 sec
2013-09-18 01:04:54,907 Stage-2 map = 92%, reduce = 0%, Cumulative CPU 81.11 sec
2013-09-18 01:04:55,912 Stage-2 map = 92%, reduce = 0%, Cumulative CPU 81.11 sec
2013-09-18 01:04:56,917 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 100.5 sec
2013-09-18 01:04:57,927 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 100.5 sec
2013-09-18 01:04:58,931 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 100.5 sec
2013-09-18 01:04:59,935 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 100.5 sec
2013-09-18 01:05:00,939 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 100.5 sec
2013-09-18 01:05:01,943 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 100.5 sec
2013-09-18 01:05:02,947 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 100.5 sec
2013-09-18 01:05:03,952 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 100.5 sec
2013-09-18 01:05:04,956 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 100.5 sec
2013-09-18 01:05:05,960 Stage-2 map = 100%, reduce = 67%, Cumulative CPU 100.5 sec
2013-09-18 01:05:06,965 Stage-2 map = 100%, reduce = 67%, Cumulative CPU 100.5 sec
2013-09-18 01:05:07,969 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 113.63 sec
2013-09-18 01:05:08,973 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 113.63 sec
MapReduce Total cumulative CPU time: 1 minutes 53 seconds 630 msec
Ended Job = job_201309172235_0140
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 11 Cumulative CPU: 697.41 sec HDFS Read: 1046532092 HDFS Write: 545445520 SUCCESS
Job 1: Map: 2 Reduce: 1 Cumulative CPU: 113.63 sec HDFS Read: 545449242 HDFS Write: 418 SUCCESS
Total MapReduce CPU Time Spent: 13 minutes 31 seconds 40 msec
OK
Time taken: 124.002 seconds, Fetched: 10 row(s)
hive> quit;
times: 2
query: SELECT WatchID, ClientIP, count(*) AS c, sum(Refresh), avg(ResolutionWidth) FROM hits_100m WHERE SearchPhrase != '' GROUP BY WatchID, ClientIP ORDER BY c DESC LIMIT 10;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_24725@mturlrep13_201309180105_1494771182.txt
hive> SELECT WatchID, ClientIP, count(*) AS c, sum(Refresh), avg(ResolutionWidth) FROM hits_100m WHERE SearchPhrase != '' GROUP BY WatchID, ClientIP ORDER BY c DESC LIMIT 10;;
Total MapReduce jobs = 2
Launching Job 1 out of 2
Number of reduce tasks not specified. Estimated from input data size: 11
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0141
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 11
2013-09-18 01:05:19,764 Stage-1 map = 0%, reduce = 0%
2013-09-18 01:05:29,107 Stage-1 map = 1%, reduce = 0%
2013-09-18 01:05:30,113 Stage-1 map = 4%, reduce = 0%
2013-09-18 01:05:31,124 Stage-1 map = 5%, reduce = 0%
2013-09-18 01:05:32,129 Stage-1 map = 12%, reduce = 0%
2013-09-18 01:05:33,137 Stage-1 map = 20%, reduce = 0%
2013-09-18 01:05:34,151 Stage-1 map = 22%, reduce = 0%
2013-09-18 01:05:35,158 Stage-1 map = 36%, reduce = 0%
2013-09-18 01:05:36,163 Stage-1 map = 39%, reduce = 0%
2013-09-18 01:05:37,173 Stage-1 map = 43%, reduce = 0%
2013-09-18 01:05:38,187 Stage-1 map = 61%, reduce = 0%, Cumulative CPU 34.53 sec
2013-09-18 01:05:39,204 Stage-1 map = 68%, reduce = 0%, Cumulative CPU 108.67 sec
2013-09-18 01:05:40,224 Stage-1 map = 75%, reduce = 0%, Cumulative CPU 150.31 sec
2013-09-18 01:05:41,238 Stage-1 map = 84%, reduce = 0%, Cumulative CPU 497.43 sec
2013-09-18 01:05:42,253 Stage-1 map = 89%, reduce = 0%, Cumulative CPU 506.05 sec
2013-09-18 01:05:43,260 Stage-1 map = 95%, reduce = 0%, Cumulative CPU 533.06 sec
2013-09-18 01:05:44,266 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 577.12 sec
2013-09-18 01:05:45,271 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 577.12 sec
2013-09-18 01:05:46,277 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 577.12 sec
2013-09-18 01:05:47,282 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 577.12 sec
2013-09-18 01:05:48,288 Stage-1 map = 100%, reduce = 8%, Cumulative CPU 577.12 sec
2013-09-18 01:05:49,309 Stage-1 map = 100%, reduce = 17%, Cumulative CPU 577.12 sec
2013-09-18 01:05:50,315 Stage-1 map = 100%, reduce = 32%, Cumulative CPU 577.12 sec
2013-09-18 01:05:51,322 Stage-1 map = 100%, reduce = 43%, Cumulative CPU 577.12 sec
2013-09-18 01:05:52,329 Stage-1 map = 100%, reduce = 49%, Cumulative CPU 577.12 sec
2013-09-18 01:05:53,335 Stage-1 map = 100%, reduce = 67%, Cumulative CPU 577.12 sec
2013-09-18 01:05:54,342 Stage-1 map = 100%, reduce = 71%, Cumulative CPU 577.12 sec
2013-09-18 01:05:55,349 Stage-1 map = 100%, reduce = 74%, Cumulative CPU 577.12 sec
2013-09-18 01:05:56,355 Stage-1 map = 100%, reduce = 82%, Cumulative CPU 577.12 sec
2013-09-18 01:05:57,364 Stage-1 map = 100%, reduce = 87%, Cumulative CPU 590.71 sec
2013-09-18 01:05:58,370 Stage-1 map = 100%, reduce = 91%, Cumulative CPU 646.07 sec
2013-09-18 01:05:59,388 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 729.12 sec
2013-09-18 01:06:00,394 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 729.12 sec
2013-09-18 01:06:01,400 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 729.12 sec
MapReduce Total cumulative CPU time: 12 minutes 9 seconds 120 msec
Ended Job = job_201309172235_0141
Launching Job 2 out of 2
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0142
Hadoop job information for Stage-2: number of mappers: 2; number of reducers: 1
2013-09-18 01:06:03,914 Stage-2 map = 0%, reduce = 0%
2013-09-18 01:06:14,948 Stage-2 map = 8%, reduce = 0%
2013-09-18 01:06:17,957 Stage-2 map = 18%, reduce = 0%
2013-09-18 01:06:20,967 Stage-2 map = 37%, reduce = 0%
2013-09-18 01:06:26,985 Stage-2 map = 55%, reduce = 0%
2013-09-18 01:06:33,003 Stage-2 map = 73%, reduce = 0%
2013-09-18 01:06:39,021 Stage-2 map = 82%, reduce = 0%
2013-09-18 01:06:44,041 Stage-2 map = 92%, reduce = 0%, Cumulative CPU 87.48 sec
2013-09-18 01:06:45,046 Stage-2 map = 92%, reduce = 0%, Cumulative CPU 87.48 sec
2013-09-18 01:06:46,051 Stage-2 map = 92%, reduce = 0%, Cumulative CPU 87.48 sec
2013-09-18 01:06:47,056 Stage-2 map = 92%, reduce = 0%, Cumulative CPU 87.48 sec
2013-09-18 01:06:48,061 Stage-2 map = 92%, reduce = 0%, Cumulative CPU 87.48 sec
2013-09-18 01:06:49,066 Stage-2 map = 92%, reduce = 0%, Cumulative CPU 87.48 sec
2013-09-18 01:06:50,071 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 99.29 sec
2013-09-18 01:06:51,076 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 99.29 sec
2013-09-18 01:06:52,092 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 99.29 sec
2013-09-18 01:06:53,097 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 99.29 sec
2013-09-18 01:06:54,101 Stage-2 map = 100%, reduce = 17%, Cumulative CPU 99.29 sec
2013-09-18 01:06:55,106 Stage-2 map = 100%, reduce = 17%, Cumulative CPU 99.29 sec
2013-09-18 01:06:56,110 Stage-2 map = 100%, reduce = 17%, Cumulative CPU 99.29 sec
2013-09-18 01:06:57,115 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 99.29 sec
2013-09-18 01:06:58,119 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 99.29 sec
2013-09-18 01:06:59,123 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 99.29 sec
2013-09-18 01:07:00,127 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 99.29 sec
2013-09-18 01:07:01,132 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 99.29 sec
2013-09-18 01:07:02,136 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 99.29 sec
2013-09-18 01:07:03,140 Stage-2 map = 100%, reduce = 67%, Cumulative CPU 99.29 sec
2013-09-18 01:07:04,144 Stage-2 map = 100%, reduce = 67%, Cumulative CPU 99.29 sec
2013-09-18 01:07:05,148 Stage-2 map = 100%, reduce = 67%, Cumulative CPU 99.29 sec
2013-09-18 01:07:06,154 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 112.67 sec
2013-09-18 01:07:07,159 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 112.67 sec
2013-09-18 01:07:08,164 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 112.67 sec
MapReduce Total cumulative CPU time: 1 minutes 52 seconds 670 msec
Ended Job = job_201309172235_0142
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 11 Cumulative CPU: 729.12 sec HDFS Read: 1046532092 HDFS Write: 545445520 SUCCESS
Job 1: Map: 2 Reduce: 1 Cumulative CPU: 112.67 sec HDFS Read: 545449242 HDFS Write: 418 SUCCESS
Total MapReduce CPU Time Spent: 14 minutes 1 seconds 790 msec
OK
Time taken: 117.366 seconds, Fetched: 10 row(s)
hive> quit;
times: 3
query: SELECT WatchID, ClientIP, count(*) AS c, sum(Refresh), avg(ResolutionWidth) FROM hits_100m WHERE SearchPhrase != '' GROUP BY WatchID, ClientIP ORDER BY c DESC LIMIT 10;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_583@mturlrep13_201309180107_219916780.txt
hive> SELECT WatchID, ClientIP, count(*) AS c, sum(Refresh), avg(ResolutionWidth) FROM hits_100m WHERE SearchPhrase != '' GROUP BY WatchID, ClientIP ORDER BY c DESC LIMIT 10;;
Total MapReduce jobs = 2
Launching Job 1 out of 2
Number of reduce tasks not specified. Estimated from input data size: 11
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0143
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 11
2013-09-18 01:07:18,186 Stage-1 map = 0%, reduce = 0%
2013-09-18 01:07:29,269 Stage-1 map = 4%, reduce = 0%
2013-09-18 01:07:30,388 Stage-1 map = 9%, reduce = 0%
2013-09-18 01:07:31,400 Stage-1 map = 11%, reduce = 0%
2013-09-18 01:07:32,410 Stage-1 map = 19%, reduce = 0%
2013-09-18 01:07:33,417 Stage-1 map = 20%, reduce = 0%
2013-09-18 01:07:34,436 Stage-1 map = 36%, reduce = 0%
2013-09-18 01:07:35,442 Stage-1 map = 39%, reduce = 0%
2013-09-18 01:07:36,448 Stage-1 map = 42%, reduce = 0%
2013-09-18 01:07:37,463 Stage-1 map = 64%, reduce = 0%, Cumulative CPU 36.86 sec
2013-09-18 01:07:38,488 Stage-1 map = 68%, reduce = 0%, Cumulative CPU 89.65 sec
2013-09-18 01:07:39,524 Stage-1 map = 76%, reduce = 0%, Cumulative CPU 170.2 sec
2013-09-18 01:07:40,544 Stage-1 map = 86%, reduce = 0%, Cumulative CPU 229.86 sec
2013-09-18 01:07:41,573 Stage-1 map = 87%, reduce = 0%, Cumulative CPU 524.54 sec
2013-09-18 01:07:42,584 Stage-1 map = 93%, reduce = 0%, Cumulative CPU 543.69 sec
2013-09-18 01:07:43,591 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 583.3 sec
2013-09-18 01:07:44,597 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 583.3 sec
2013-09-18 01:07:45,602 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 583.3 sec
2013-09-18 01:07:46,609 Stage-1 map = 100%, reduce = 3%, Cumulative CPU 583.3 sec
2013-09-18 01:07:47,617 Stage-1 map = 100%, reduce = 6%, Cumulative CPU 583.3 sec
2013-09-18 01:07:48,627 Stage-1 map = 100%, reduce = 23%, Cumulative CPU 583.3 sec
2013-09-18 01:07:49,634 Stage-1 map = 100%, reduce = 35%, Cumulative CPU 583.3 sec
2013-09-18 01:07:50,641 Stage-1 map = 100%, reduce = 38%, Cumulative CPU 583.3 sec
2013-09-18 01:07:51,648 Stage-1 map = 100%, reduce = 55%, Cumulative CPU 583.3 sec
2013-09-18 01:07:52,655 Stage-1 map = 100%, reduce = 68%, Cumulative CPU 583.3 sec
2013-09-18 01:07:53,662 Stage-1 map = 100%, reduce = 70%, Cumulative CPU 583.3 sec
2013-09-18 01:07:54,669 Stage-1 map = 100%, reduce = 77%, Cumulative CPU 583.3 sec
2013-09-18 01:07:55,676 Stage-1 map = 100%, reduce = 84%, Cumulative CPU 583.3 sec
2013-09-18 01:07:56,686 Stage-1 map = 100%, reduce = 86%, Cumulative CPU 610.88 sec
2013-09-18 01:07:57,693 Stage-1 map = 100%, reduce = 94%, Cumulative CPU 665.19 sec
2013-09-18 01:07:58,698 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 735.33 sec
2013-09-18 01:07:59,704 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 735.33 sec
MapReduce Total cumulative CPU time: 12 minutes 15 seconds 330 msec
Ended Job = job_201309172235_0143
Launching Job 2 out of 2
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0144
Hadoop job information for Stage-2: number of mappers: 2; number of reducers: 1
2013-09-18 01:08:02,196 Stage-2 map = 0%, reduce = 0%
2013-09-18 01:08:13,229 Stage-2 map = 18%, reduce = 0%
2013-09-18 01:08:19,247 Stage-2 map = 37%, reduce = 0%
2013-09-18 01:08:26,358 Stage-2 map = 55%, reduce = 0%
2013-09-18 01:08:31,375 Stage-2 map = 73%, reduce = 0%
2013-09-18 01:08:34,385 Stage-2 map = 82%, reduce = 0%
2013-09-18 01:08:40,407 Stage-2 map = 92%, reduce = 0%, Cumulative CPU 45.23 sec
2013-09-18 01:08:41,412 Stage-2 map = 92%, reduce = 0%, Cumulative CPU 90.45 sec
2013-09-18 01:08:42,416 Stage-2 map = 92%, reduce = 0%, Cumulative CPU 90.45 sec
2013-09-18 01:08:43,421 Stage-2 map = 92%, reduce = 0%, Cumulative CPU 90.45 sec
2013-09-18 01:08:44,425 Stage-2 map = 92%, reduce = 0%, Cumulative CPU 90.45 sec
2013-09-18 01:08:45,430 Stage-2 map = 92%, reduce = 0%, Cumulative CPU 90.45 sec
2013-09-18 01:08:46,435 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 96.92 sec
2013-09-18 01:08:47,441 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 96.92 sec
2013-09-18 01:08:48,446 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 96.92 sec
2013-09-18 01:08:49,451 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 96.92 sec
2013-09-18 01:08:50,457 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 96.92 sec
2013-09-18 01:08:51,461 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 96.92 sec
2013-09-18 01:08:52,467 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 96.92 sec
2013-09-18 01:08:53,472 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 96.92 sec
2013-09-18 01:08:54,477 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 96.92 sec
2013-09-18 01:08:55,482 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 96.92 sec
2013-09-18 01:08:56,488 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 96.92 sec
2013-09-18 01:08:57,499 Stage-2 map = 100%, reduce = 67%, Cumulative CPU 96.92 sec
2013-09-18 01:08:58,504 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 109.55 sec
2013-09-18 01:08:59,510 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 109.55 sec
MapReduce Total cumulative CPU time: 1 minutes 49 seconds 550 msec
Ended Job = job_201309172235_0144
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 11 Cumulative CPU: 735.33 sec HDFS Read: 1046532092 HDFS Write: 545445520 SUCCESS
Job 1: Map: 2 Reduce: 1 Cumulative CPU: 109.55 sec HDFS Read: 545449242 HDFS Write: 418 SUCCESS
Total MapReduce CPU Time Spent: 14 minutes 4 seconds 880 msec
OK
Time taken: 109.524 seconds, Fetched: 10 row(s)
hive> quit;
-- агрегация по двум полям, которая ничего не агрегирует. Для больших таблиц выполнить не получится.;
times: 1
query: SELECT WatchID, ClientIP, count(*) AS c, sum(Refresh), avg(ResolutionWidth) FROM hits_100m GROUP BY WatchID, ClientIP ORDER BY c DESC LIMIT 10;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_9002@mturlrep13_201309180109_1127417909.txt
hive> SELECT WatchID, ClientIP, count(*) AS c, sum(Refresh), avg(ResolutionWidth) FROM hits_100m GROUP BY WatchID, ClientIP ORDER BY c DESC LIMIT 10;;
Total MapReduce jobs = 2
Launching Job 1 out of 2
Number of reduce tasks not specified. Estimated from input data size: 11
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0145
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 11
2013-09-18 01:09:21,157 Stage-1 map = 0%, reduce = 0%
2013-09-18 01:09:34,238 Stage-1 map = 1%, reduce = 0%
2013-09-18 01:09:37,271 Stage-1 map = 2%, reduce = 0%
2013-09-18 01:09:38,281 Stage-1 map = 5%, reduce = 0%
2013-09-18 01:09:39,287 Stage-1 map = 6%, reduce = 0%
2013-09-18 01:09:40,293 Stage-1 map = 11%, reduce = 0%
2013-09-18 01:09:41,307 Stage-1 map = 15%, reduce = 0%
2013-09-18 01:09:42,344 Stage-1 map = 16%, reduce = 0%, Cumulative CPU 424.15 sec
2013-09-18 01:09:43,369 Stage-1 map = 18%, reduce = 0%, Cumulative CPU 424.15 sec
2013-09-18 01:09:44,396 Stage-1 map = 21%, reduce = 0%, Cumulative CPU 424.15 sec
2013-09-18 01:09:45,410 Stage-1 map = 22%, reduce = 0%, Cumulative CPU 424.15 sec
2013-09-18 01:09:46,422 Stage-1 map = 26%, reduce = 0%, Cumulative CPU 424.15 sec
2013-09-18 01:09:47,429 Stage-1 map = 32%, reduce = 0%, Cumulative CPU 424.15 sec
2013-09-18 01:09:48,440 Stage-1 map = 34%, reduce = 0%, Cumulative CPU 424.15 sec
2013-09-18 01:09:49,454 Stage-1 map = 36%, reduce = 0%, Cumulative CPU 424.15 sec
2013-09-18 01:09:50,470 Stage-1 map = 43%, reduce = 0%, Cumulative CPU 424.15 sec
2013-09-18 01:09:51,477 Stage-1 map = 45%, reduce = 0%, Cumulative CPU 424.15 sec
2013-09-18 01:09:52,487 Stage-1 map = 47%, reduce = 0%, Cumulative CPU 424.15 sec
2013-09-18 01:09:53,494 Stage-1 map = 52%, reduce = 0%, Cumulative CPU 424.15 sec
2013-09-18 01:09:54,510 Stage-1 map = 55%, reduce = 0%, Cumulative CPU 424.15 sec
2013-09-18 01:09:55,539 Stage-1 map = 56%, reduce = 0%, Cumulative CPU 424.15 sec
2013-09-18 01:09:56,564 Stage-1 map = 63%, reduce = 0%, Cumulative CPU 424.15 sec
2013-09-18 01:09:57,574 Stage-1 map = 65%, reduce = 0%, Cumulative CPU 424.15 sec
2013-09-18 01:09:58,586 Stage-1 map = 67%, reduce = 0%, Cumulative CPU 424.15 sec
2013-09-18 01:09:59,593 Stage-1 map = 72%, reduce = 0%, Cumulative CPU 424.15 sec
2013-09-18 01:10:00,605 Stage-1 map = 76%, reduce = 0%, Cumulative CPU 424.15 sec
2013-09-18 01:10:01,613 Stage-1 map = 76%, reduce = 0%, Cumulative CPU 424.15 sec
2013-09-18 01:10:02,631 Stage-1 map = 81%, reduce = 0%, Cumulative CPU 461.38 sec
2013-09-18 01:10:03,659 Stage-1 map = 83%, reduce = 0%, Cumulative CPU 461.38 sec
2013-09-18 01:10:04,670 Stage-1 map = 84%, reduce = 0%, Cumulative CPU 504.24 sec
2013-09-18 01:10:05,681 Stage-1 map = 85%, reduce = 0%, Cumulative CPU 504.24 sec
2013-09-18 01:10:06,688 Stage-1 map = 87%, reduce = 0%, Cumulative CPU 526.72 sec
2013-09-18 01:10:07,709 Stage-1 map = 89%, reduce = 0%, Cumulative CPU 618.33 sec
2013-09-18 01:10:08,722 Stage-1 map = 91%, reduce = 0%, Cumulative CPU 708.66 sec
2013-09-18 01:10:09,737 Stage-1 map = 94%, reduce = 0%, Cumulative CPU 906.45 sec
2013-09-18 01:10:10,743 Stage-1 map = 95%, reduce = 0%, Cumulative CPU 956.72 sec
2013-09-18 01:10:11,750 Stage-1 map = 97%, reduce = 0%, Cumulative CPU 1148.71 sec
2013-09-18 01:10:12,756 Stage-1 map = 99%, reduce = 0%, Cumulative CPU 1258.63 sec
2013-09-18 01:10:13,762 Stage-1 map = 99%, reduce = 0%, Cumulative CPU 1258.63 sec
2013-09-18 01:10:14,770 Stage-1 map = 99%, reduce = 0%, Cumulative CPU 1288.05 sec
2013-09-18 01:10:15,777 Stage-1 map = 100%, reduce = 2%, Cumulative CPU 1356.32 sec
2013-09-18 01:10:16,784 Stage-1 map = 100%, reduce = 3%, Cumulative CPU 1356.32 sec
2013-09-18 01:10:17,807 Stage-1 map = 100%, reduce = 9%, Cumulative CPU 1356.32 sec
2013-09-18 01:10:18,813 Stage-1 map = 100%, reduce = 21%, Cumulative CPU 1356.32 sec
2013-09-18 01:10:19,819 Stage-1 map = 100%, reduce = 26%, Cumulative CPU 1356.32 sec
2013-09-18 01:10:20,825 Stage-1 map = 100%, reduce = 32%, Cumulative CPU 1356.32 sec
2013-09-18 01:10:21,831 Stage-1 map = 100%, reduce = 32%, Cumulative CPU 1356.32 sec
2013-09-18 01:10:22,837 Stage-1 map = 100%, reduce = 32%, Cumulative CPU 1356.32 sec
2013-09-18 01:10:26,141 Stage-1 map = 100%, reduce = 45%, Cumulative CPU 1356.32 sec
2013-09-18 01:10:27,147 Stage-1 map = 100%, reduce = 45%, Cumulative CPU 1356.32 sec
2013-09-18 01:10:28,153 Stage-1 map = 100%, reduce = 48%, Cumulative CPU 1356.32 sec
2013-09-18 01:10:29,328 Stage-1 map = 100%, reduce = 58%, Cumulative CPU 1356.32 sec
2013-09-18 01:10:30,334 Stage-1 map = 100%, reduce = 58%, Cumulative CPU 1356.32 sec
2013-09-18 01:10:31,340 Stage-1 map = 100%, reduce = 58%, Cumulative CPU 1356.32 sec
2013-09-18 01:10:32,346 Stage-1 map = 100%, reduce = 58%, Cumulative CPU 1356.32 sec
2013-09-18 01:10:33,352 Stage-1 map = 100%, reduce = 58%, Cumulative CPU 1356.32 sec
2013-09-18 01:10:34,359 Stage-1 map = 100%, reduce = 68%, Cumulative CPU 1356.32 sec
2013-09-18 01:10:36,108 Stage-1 map = 100%, reduce = 68%, Cumulative CPU 1356.32 sec
2013-09-18 01:10:37,114 Stage-1 map = 100%, reduce = 68%, Cumulative CPU 1356.32 sec
2013-09-18 01:10:38,121 Stage-1 map = 100%, reduce = 68%, Cumulative CPU 1356.32 sec
2013-09-18 01:10:39,470 Stage-1 map = 100%, reduce = 69%, Cumulative CPU 1356.32 sec
2013-09-18 01:10:40,477 Stage-1 map = 100%, reduce = 70%, Cumulative CPU 1356.32 sec
2013-09-18 01:10:41,483 Stage-1 map = 100%, reduce = 71%, Cumulative CPU 1356.32 sec
2013-09-18 01:10:42,629 Stage-1 map = 100%, reduce = 71%, Cumulative CPU 1569.69 sec
2013-09-18 01:10:45,862 Stage-1 map = 100%, reduce = 72%, Cumulative CPU 1569.69 sec
2013-09-18 01:10:48,964 Stage-1 map = 100%, reduce = 73%, Cumulative CPU 1569.69 sec
2013-09-18 01:10:49,969 Stage-1 map = 100%, reduce = 73%, Cumulative CPU 1569.69 sec
2013-09-18 01:10:50,976 Stage-1 map = 100%, reduce = 73%, Cumulative CPU 1569.69 sec
2013-09-18 01:10:51,982 Stage-1 map = 100%, reduce = 74%, Cumulative CPU 1569.69 sec
2013-09-18 01:10:52,988 Stage-1 map = 100%, reduce = 74%, Cumulative CPU 1569.69 sec
2013-09-18 01:10:53,993 Stage-1 map = 100%, reduce = 74%, Cumulative CPU 1569.69 sec
2013-09-18 01:10:54,999 Stage-1 map = 100%, reduce = 77%, Cumulative CPU 1569.69 sec
2013-09-18 01:10:56,005 Stage-1 map = 100%, reduce = 77%, Cumulative CPU 1569.69 sec
2013-09-18 01:10:57,010 Stage-1 map = 100%, reduce = 77%, Cumulative CPU 1569.69 sec
2013-09-18 01:10:58,016 Stage-1 map = 100%, reduce = 78%, Cumulative CPU 1569.69 sec
2013-09-18 01:10:59,021 Stage-1 map = 100%, reduce = 78%, Cumulative CPU 1569.69 sec
2013-09-18 01:11:00,027 Stage-1 map = 100%, reduce = 78%, Cumulative CPU 1569.69 sec
2013-09-18 01:11:01,033 Stage-1 map = 100%, reduce = 81%, Cumulative CPU 1569.69 sec
2013-09-18 01:11:02,038 Stage-1 map = 100%, reduce = 81%, Cumulative CPU 1569.69 sec
2013-09-18 01:11:03,043 Stage-1 map = 100%, reduce = 81%, Cumulative CPU 1569.69 sec
2013-09-18 01:11:04,049 Stage-1 map = 100%, reduce = 83%, Cumulative CPU 1569.69 sec
2013-09-18 01:11:06,499 Stage-1 map = 100%, reduce = 83%, Cumulative CPU 1569.69 sec
2013-09-18 01:11:07,504 Stage-1 map = 100%, reduce = 83%, Cumulative CPU 1569.69 sec
2013-09-18 01:11:08,509 Stage-1 map = 100%, reduce = 83%, Cumulative CPU 1569.69 sec
2013-09-18 01:11:09,513 Stage-1 map = 100%, reduce = 83%, Cumulative CPU 1569.69 sec
2013-09-18 01:11:10,519 Stage-1 map = 100%, reduce = 83%, Cumulative CPU 1569.69 sec
2013-09-18 01:11:12,086 Stage-1 map = 100%, reduce = 83%, Cumulative CPU 1569.69 sec
2013-09-18 01:11:13,091 Stage-1 map = 100%, reduce = 83%, Cumulative CPU 1569.69 sec
2013-09-18 01:11:14,097 Stage-1 map = 100%, reduce = 83%, Cumulative CPU 1569.69 sec
2013-09-18 01:11:15,102 Stage-1 map = 100%, reduce = 83%, Cumulative CPU 1569.69 sec
2013-09-18 01:11:16,108 Stage-1 map = 100%, reduce = 85%, Cumulative CPU 1569.69 sec
2013-09-18 01:11:17,635 Stage-1 map = 100%, reduce = 85%, Cumulative CPU 1569.69 sec
2013-09-18 01:11:18,641 Stage-1 map = 100%, reduce = 85%, Cumulative CPU 1569.69 sec
2013-09-18 01:11:21,528 Stage-1 map = 100%, reduce = 87%, Cumulative CPU 1569.69 sec
2013-09-18 01:11:24,908 Stage-1 map = 100%, reduce = 89%, Cumulative CPU 1569.69 sec
2013-09-18 01:11:25,914 Stage-1 map = 100%, reduce = 89%, Cumulative CPU 1569.69 sec
2013-09-18 01:11:26,919 Stage-1 map = 100%, reduce = 89%, Cumulative CPU 1569.69 sec
2013-09-18 01:11:27,926 Stage-1 map = 100%, reduce = 91%, Cumulative CPU 1569.69 sec
2013-09-18 01:11:28,931 Stage-1 map = 100%, reduce = 91%, Cumulative CPU 1569.69 sec
2013-09-18 01:11:29,959 Stage-1 map = 100%, reduce = 91%, Cumulative CPU 1569.69 sec
2013-09-18 01:11:30,966 Stage-1 map = 100%, reduce = 94%, Cumulative CPU 1569.69 sec
2013-09-18 01:11:31,971 Stage-1 map = 100%, reduce = 94%, Cumulative CPU 1569.69 sec
2013-09-18 01:11:33,303 Stage-1 map = 100%, reduce = 95%, Cumulative CPU 1569.69 sec
2013-09-18 01:11:34,309 Stage-1 map = 100%, reduce = 95%, Cumulative CPU 1569.69 sec
2013-09-18 01:11:35,315 Stage-1 map = 100%, reduce = 95%, Cumulative CPU 1569.69 sec
2013-09-18 01:11:36,321 Stage-1 map = 100%, reduce = 97%, Cumulative CPU 1607.57 sec
2013-09-18 01:11:37,327 Stage-1 map = 100%, reduce = 97%, Cumulative CPU 1607.57 sec
2013-09-18 01:11:38,336 Stage-1 map = 100%, reduce = 97%, Cumulative CPU 1728.83 sec
2013-09-18 01:11:39,342 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1812.48 sec
2013-09-18 01:11:40,348 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1812.48 sec
2013-09-18 01:11:41,354 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1812.48 sec
2013-09-18 01:11:42,360 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 2013.45 sec
2013-09-18 01:11:43,365 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 2030.96 sec
2013-09-18 01:11:44,371 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 2030.96 sec
MapReduce Total cumulative CPU time: 33 minutes 50 seconds 960 msec
Ended Job = job_201309172235_0145
Launching Job 2 out of 2
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0146
Hadoop job information for Stage-2: number of mappers: 14; number of reducers: 1
2013-09-18 01:11:47,866 Stage-2 map = 0%, reduce = 0%
2013-09-18 01:12:01,932 Stage-2 map = 2%, reduce = 0%
2013-09-18 01:12:04,943 Stage-2 map = 15%, reduce = 0%
2013-09-18 01:12:07,952 Stage-2 map = 19%, reduce = 0%
2013-09-18 01:12:09,062 Stage-2 map = 22%, reduce = 0%
2013-09-18 01:12:11,068 Stage-2 map = 26%, reduce = 0%
2013-09-18 01:12:15,081 Stage-2 map = 28%, reduce = 0%
2013-09-18 01:12:17,087 Stage-2 map = 34%, reduce = 0%
2013-09-18 01:12:18,091 Stage-2 map = 43%, reduce = 0%
2013-09-18 01:12:21,102 Stage-2 map = 48%, reduce = 0%
2013-09-18 01:12:23,108 Stage-2 map = 52%, reduce = 0%
2013-09-18 01:12:24,112 Stage-2 map = 64%, reduce = 0%
2013-09-18 01:12:30,131 Stage-2 map = 73%, reduce = 0%
2013-09-18 01:12:33,143 Stage-2 map = 78%, reduce = 0%
2013-09-18 01:12:35,159 Stage-2 map = 79%, reduce = 0%, Cumulative CPU 54.68 sec
2013-09-18 01:12:36,935 Stage-2 map = 79%, reduce = 0%, Cumulative CPU 54.68 sec
2013-09-18 01:12:37,940 Stage-2 map = 79%, reduce = 0%, Cumulative CPU 54.68 sec
2013-09-18 01:12:38,945 Stage-2 map = 81%, reduce = 0%, Cumulative CPU 111.92 sec
2013-09-18 01:12:39,950 Stage-2 map = 85%, reduce = 0%, Cumulative CPU 231.05 sec
2013-09-18 01:12:40,956 Stage-2 map = 85%, reduce = 0%, Cumulative CPU 231.05 sec
2013-09-18 01:12:41,961 Stage-2 map = 85%, reduce = 0%, Cumulative CPU 231.05 sec
2013-09-18 01:12:42,966 Stage-2 map = 85%, reduce = 0%, Cumulative CPU 842.33 sec
2013-09-18 01:12:43,971 Stage-2 map = 85%, reduce = 0%, Cumulative CPU 842.33 sec
2013-09-18 01:12:45,166 Stage-2 map = 85%, reduce = 0%, Cumulative CPU 842.33 sec
2013-09-18 01:12:46,171 Stage-2 map = 86%, reduce = 10%, Cumulative CPU 845.72 sec
2013-09-18 01:12:47,176 Stage-2 map = 95%, reduce = 10%, Cumulative CPU 870.78 sec
2013-09-18 01:12:48,180 Stage-2 map = 100%, reduce = 10%, Cumulative CPU 887.76 sec
2013-09-18 01:12:49,184 Stage-2 map = 100%, reduce = 10%, Cumulative CPU 887.76 sec
2013-09-18 01:12:50,188 Stage-2 map = 100%, reduce = 10%, Cumulative CPU 887.76 sec
2013-09-18 01:12:53,214 Stage-2 map = 100%, reduce = 10%, Cumulative CPU 887.76 sec
2013-09-18 01:12:55,469 Stage-2 map = 100%, reduce = 14%, Cumulative CPU 887.76 sec
2013-09-18 01:12:56,473 Stage-2 map = 100%, reduce = 14%, Cumulative CPU 887.76 sec
2013-09-18 01:12:57,477 Stage-2 map = 100%, reduce = 14%, Cumulative CPU 887.76 sec
2013-09-18 01:12:58,480 Stage-2 map = 100%, reduce = 14%, Cumulative CPU 887.76 sec
2013-09-18 01:12:59,485 Stage-2 map = 100%, reduce = 14%, Cumulative CPU 887.76 sec
2013-09-18 01:13:00,557 Stage-2 map = 100%, reduce = 14%, Cumulative CPU 887.76 sec
2013-09-18 01:13:01,561 Stage-2 map = 100%, reduce = 14%, Cumulative CPU 887.76 sec
2013-09-18 01:13:02,566 Stage-2 map = 100%, reduce = 14%, Cumulative CPU 887.76 sec
2013-09-18 01:13:03,571 Stage-2 map = 100%, reduce = 17%, Cumulative CPU 887.76 sec
2013-09-18 01:13:04,575 Stage-2 map = 100%, reduce = 17%, Cumulative CPU 887.76 sec
2013-09-18 01:13:05,579 Stage-2 map = 100%, reduce = 17%, Cumulative CPU 887.76 sec
2013-09-18 01:13:06,583 Stage-2 map = 100%, reduce = 17%, Cumulative CPU 887.76 sec
2013-09-18 01:13:10,271 Stage-2 map = 100%, reduce = 19%, Cumulative CPU 887.76 sec
2013-09-18 01:13:11,275 Stage-2 map = 100%, reduce = 19%, Cumulative CPU 887.76 sec
2013-09-18 01:13:12,279 Stage-2 map = 100%, reduce = 19%, Cumulative CPU 887.76 sec
2013-09-18 01:13:13,283 Stage-2 map = 100%, reduce = 24%, Cumulative CPU 887.76 sec
2013-09-18 01:13:14,287 Stage-2 map = 100%, reduce = 24%, Cumulative CPU 887.76 sec
2013-09-18 01:13:15,291 Stage-2 map = 100%, reduce = 24%, Cumulative CPU 887.76 sec
2013-09-18 01:13:16,295 Stage-2 map = 100%, reduce = 24%, Cumulative CPU 887.76 sec
2013-09-18 01:13:17,544 Stage-2 map = 100%, reduce = 24%, Cumulative CPU 887.76 sec
2013-09-18 01:13:18,548 Stage-2 map = 100%, reduce = 24%, Cumulative CPU 887.76 sec
2013-09-18 01:13:19,552 Stage-2 map = 100%, reduce = 24%, Cumulative CPU 887.76 sec
2013-09-18 01:13:21,226 Stage-2 map = 100%, reduce = 26%, Cumulative CPU 887.76 sec
2013-09-18 01:13:22,398 Stage-2 map = 100%, reduce = 26%, Cumulative CPU 887.76 sec
2013-09-18 01:13:26,427 Stage-2 map = 100%, reduce = 29%, Cumulative CPU 887.76 sec
2013-09-18 01:13:27,431 Stage-2 map = 100%, reduce = 29%, Cumulative CPU 887.76 sec
2013-09-18 01:13:28,435 Stage-2 map = 100%, reduce = 29%, Cumulative CPU 887.76 sec
2013-09-18 01:13:29,440 Stage-2 map = 100%, reduce = 31%, Cumulative CPU 887.76 sec
2013-09-18 01:13:30,447 Stage-2 map = 100%, reduce = 31%, Cumulative CPU 887.76 sec
2013-09-18 01:13:31,451 Stage-2 map = 100%, reduce = 31%, Cumulative CPU 887.76 sec
2013-09-18 01:13:32,455 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 887.76 sec
2013-09-18 01:13:33,459 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 887.76 sec
2013-09-18 01:13:34,463 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 887.76 sec
2013-09-18 01:13:35,467 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 887.76 sec
2013-09-18 01:13:36,471 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 887.76 sec
2013-09-18 01:13:37,475 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 887.76 sec
2013-09-18 01:13:38,520 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 887.76 sec
2013-09-18 01:13:39,524 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 887.76 sec
2013-09-18 01:13:40,528 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 887.76 sec
2013-09-18 01:13:41,533 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 887.76 sec
2013-09-18 01:13:42,538 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 947.26 sec
2013-09-18 01:13:43,543 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 947.26 sec
2013-09-18 01:13:44,548 Stage-2 map = 100%, reduce = 67%, Cumulative CPU 947.26 sec
2013-09-18 01:13:45,553 Stage-2 map = 100%, reduce = 67%, Cumulative CPU 947.26 sec
2013-09-18 01:13:46,557 Stage-2 map = 100%, reduce = 67%, Cumulative CPU 947.26 sec
2013-09-18 01:13:47,561 Stage-2 map = 100%, reduce = 67%, Cumulative CPU 947.26 sec
2013-09-18 01:13:48,566 Stage-2 map = 100%, reduce = 67%, Cumulative CPU 947.26 sec
2013-09-18 01:13:49,570 Stage-2 map = 100%, reduce = 67%, Cumulative CPU 947.26 sec
2013-09-18 01:13:50,574 Stage-2 map = 100%, reduce = 67%, Cumulative CPU 947.26 sec
2013-09-18 01:13:51,662 Stage-2 map = 100%, reduce = 67%, Cumulative CPU 947.26 sec
2013-09-18 01:13:52,666 Stage-2 map = 100%, reduce = 67%, Cumulative CPU 947.26 sec
2013-09-18 01:13:57,873 Stage-2 map = 100%, reduce = 67%, Cumulative CPU 947.26 sec
2013-09-18 01:13:58,877 Stage-2 map = 100%, reduce = 67%, Cumulative CPU 947.26 sec
2013-09-18 01:13:59,880 Stage-2 map = 100%, reduce = 67%, Cumulative CPU 947.26 sec
2013-09-18 01:14:00,884 Stage-2 map = 100%, reduce = 67%, Cumulative CPU 947.26 sec
2013-09-18 01:14:01,888 Stage-2 map = 100%, reduce = 67%, Cumulative CPU 947.26 sec
2013-09-18 01:14:02,892 Stage-2 map = 100%, reduce = 67%, Cumulative CPU 947.26 sec
2013-09-18 01:14:03,895 Stage-2 map = 100%, reduce = 67%, Cumulative CPU 947.26 sec
2013-09-18 01:14:04,899 Stage-2 map = 100%, reduce = 67%, Cumulative CPU 947.26 sec
2013-09-18 01:14:05,903 Stage-2 map = 100%, reduce = 67%, Cumulative CPU 947.26 sec
2013-09-18 01:14:10,178 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 976.07 sec
2013-09-18 01:14:11,182 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 976.07 sec
2013-09-18 01:14:12,187 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 976.07 sec
MapReduce Total cumulative CPU time: 16 minutes 16 seconds 70 msec
Ended Job = job_201309172235_0146
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 11 Cumulative CPU: 2030.96 sec HDFS Read: 869197327 HDFS Write: 4140713341 SUCCESS
Job 1: Map: 14 Reduce: 1 Cumulative CPU: 976.07 sec HDFS Read: 4140841812 HDFS Write: 415 SUCCESS
Total MapReduce CPU Time Spent: 50 minutes 7 seconds 30 msec
OK
Time taken: 301.77 seconds, Fetched: 10 row(s)
hive> quit;
times: 2
query: SELECT WatchID, ClientIP, count(*) AS c, sum(Refresh), avg(ResolutionWidth) FROM hits_100m GROUP BY WatchID, ClientIP ORDER BY c DESC LIMIT 10;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_19253@mturlrep13_201309180114_1319337284.txt
hive> SELECT WatchID, ClientIP, count(*) AS c, sum(Refresh), avg(ResolutionWidth) FROM hits_100m GROUP BY WatchID, ClientIP ORDER BY c DESC LIMIT 10;;
Total MapReduce jobs = 2
Launching Job 1 out of 2
Number of reduce tasks not specified. Estimated from input data size: 11
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0147
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 11
2013-09-18 01:14:21,934 Stage-1 map = 0%, reduce = 0%
2013-09-18 01:14:38,062 Stage-1 map = 3%, reduce = 0%
2013-09-18 01:14:39,067 Stage-1 map = 4%, reduce = 0%
2013-09-18 01:14:40,081 Stage-1 map = 5%, reduce = 0%
2013-09-18 01:14:41,089 Stage-1 map = 9%, reduce = 0%
2013-09-18 01:14:42,106 Stage-1 map = 14%, reduce = 0%, Cumulative CPU 456.6 sec
2013-09-18 01:14:43,120 Stage-1 map = 15%, reduce = 0%, Cumulative CPU 456.6 sec
2013-09-18 01:14:44,128 Stage-1 map = 19%, reduce = 0%, Cumulative CPU 456.6 sec
2013-09-18 01:14:45,134 Stage-1 map = 27%, reduce = 0%, Cumulative CPU 456.6 sec
2013-09-18 01:14:46,141 Stage-1 map = 27%, reduce = 0%, Cumulative CPU 456.6 sec
2013-09-18 01:14:47,152 Stage-1 map = 29%, reduce = 0%, Cumulative CPU 456.6 sec
2013-09-18 01:14:48,158 Stage-1 map = 36%, reduce = 0%, Cumulative CPU 456.6 sec
2013-09-18 01:14:49,164 Stage-1 map = 36%, reduce = 0%, Cumulative CPU 456.6 sec
2013-09-18 01:14:50,177 Stage-1 map = 38%, reduce = 0%, Cumulative CPU 456.6 sec
2013-09-18 01:14:51,183 Stage-1 map = 45%, reduce = 0%, Cumulative CPU 456.6 sec
2013-09-18 01:14:52,193 Stage-1 map = 48%, reduce = 0%, Cumulative CPU 456.6 sec
2013-09-18 01:14:53,199 Stage-1 map = 49%, reduce = 0%, Cumulative CPU 456.6 sec
2013-09-18 01:14:54,206 Stage-1 map = 56%, reduce = 0%, Cumulative CPU 456.6 sec
2013-09-18 01:14:55,212 Stage-1 map = 58%, reduce = 0%, Cumulative CPU 456.6 sec
2013-09-18 01:14:56,218 Stage-1 map = 60%, reduce = 0%, Cumulative CPU 456.6 sec
2013-09-18 01:14:57,225 Stage-1 map = 65%, reduce = 0%, Cumulative CPU 456.6 sec
2013-09-18 01:14:58,236 Stage-1 map = 68%, reduce = 0%, Cumulative CPU 456.6 sec
2013-09-18 01:14:59,256 Stage-1 map = 68%, reduce = 0%, Cumulative CPU 456.6 sec
2013-09-18 01:15:00,265 Stage-1 map = 74%, reduce = 0%, Cumulative CPU 472.96 sec
2013-09-18 01:15:01,273 Stage-1 map = 78%, reduce = 0%, Cumulative CPU 472.96 sec
2013-09-18 01:15:02,279 Stage-1 map = 79%, reduce = 0%, Cumulative CPU 472.96 sec
2013-09-18 01:15:03,289 Stage-1 map = 81%, reduce = 0%, Cumulative CPU 472.96 sec
2013-09-18 01:15:04,295 Stage-1 map = 84%, reduce = 0%, Cumulative CPU 472.96 sec
2013-09-18 01:15:05,301 Stage-1 map = 87%, reduce = 0%, Cumulative CPU 535.78 sec
2013-09-18 01:15:06,312 Stage-1 map = 88%, reduce = 0%, Cumulative CPU 600.59 sec
2013-09-18 01:15:07,321 Stage-1 map = 90%, reduce = 0%, Cumulative CPU 709.5 sec
2013-09-18 01:15:08,347 Stage-1 map = 91%, reduce = 0%, Cumulative CPU 776.98 sec
2013-09-18 01:15:09,362 Stage-1 map = 94%, reduce = 0%, Cumulative CPU 945.32 sec
2013-09-18 01:15:10,368 Stage-1 map = 96%, reduce = 0%, Cumulative CPU 1072.86 sec
2013-09-18 01:15:11,374 Stage-1 map = 97%, reduce = 0%, Cumulative CPU 1125.81 sec
2013-09-18 01:15:12,400 Stage-1 map = 99%, reduce = 0%, Cumulative CPU 1285.77 sec
2013-09-18 01:15:13,406 Stage-1 map = 99%, reduce = 0%, Cumulative CPU 1285.77 sec
2013-09-18 01:15:14,411 Stage-1 map = 99%, reduce = 0%, Cumulative CPU 1285.77 sec
2013-09-18 01:15:15,418 Stage-1 map = 100%, reduce = 3%, Cumulative CPU 1349.28 sec
2013-09-18 01:15:16,425 Stage-1 map = 100%, reduce = 7%, Cumulative CPU 1349.28 sec
2013-09-18 01:15:17,452 Stage-1 map = 100%, reduce = 8%, Cumulative CPU 1349.28 sec
2013-09-18 01:15:18,458 Stage-1 map = 100%, reduce = 17%, Cumulative CPU 1349.28 sec
2013-09-18 01:15:19,463 Stage-1 map = 100%, reduce = 31%, Cumulative CPU 1349.28 sec
2013-09-18 01:15:20,469 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 1349.28 sec
2013-09-18 01:15:21,475 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 1349.28 sec
2013-09-18 01:15:23,349 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 1349.28 sec
2013-09-18 01:15:24,354 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 1349.28 sec
2013-09-18 01:15:25,360 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 1349.28 sec
2013-09-18 01:15:26,368 Stage-1 map = 100%, reduce = 55%, Cumulative CPU 1349.28 sec
2013-09-18 01:15:27,374 Stage-1 map = 100%, reduce = 61%, Cumulative CPU 1349.28 sec
2013-09-18 01:15:29,096 Stage-1 map = 100%, reduce = 64%, Cumulative CPU 1349.28 sec
2013-09-18 01:15:30,102 Stage-1 map = 100%, reduce = 64%, Cumulative CPU 1349.28 sec
2013-09-18 01:15:31,107 Stage-1 map = 100%, reduce = 64%, Cumulative CPU 1349.28 sec
2013-09-18 01:15:32,113 Stage-1 map = 100%, reduce = 64%, Cumulative CPU 1349.28 sec
2013-09-18 01:15:33,119 Stage-1 map = 100%, reduce = 64%, Cumulative CPU 1349.28 sec
2013-09-18 01:15:34,124 Stage-1 map = 100%, reduce = 64%, Cumulative CPU 1349.28 sec
2013-09-18 01:15:35,356 Stage-1 map = 100%, reduce = 68%, Cumulative CPU 1349.28 sec
2013-09-18 01:15:36,362 Stage-1 map = 100%, reduce = 68%, Cumulative CPU 1349.28 sec
2013-09-18 01:15:37,368 Stage-1 map = 100%, reduce = 68%, Cumulative CPU 1349.28 sec
2013-09-18 01:15:38,374 Stage-1 map = 100%, reduce = 68%, Cumulative CPU 1349.28 sec
2013-09-18 01:15:39,379 Stage-1 map = 100%, reduce = 69%, Cumulative CPU 1349.28 sec
2013-09-18 01:15:40,385 Stage-1 map = 100%, reduce = 69%, Cumulative CPU 1349.28 sec
2013-09-18 01:15:41,390 Stage-1 map = 100%, reduce = 70%, Cumulative CPU 1349.28 sec
2013-09-18 01:15:43,287 Stage-1 map = 100%, reduce = 70%, Cumulative CPU 1558.8 sec
2013-09-18 01:15:44,293 Stage-1 map = 100%, reduce = 71%, Cumulative CPU 1558.8 sec
2013-09-18 01:15:45,299 Stage-1 map = 100%, reduce = 71%, Cumulative CPU 1558.8 sec
2013-09-18 01:15:46,306 Stage-1 map = 100%, reduce = 72%, Cumulative CPU 1558.8 sec
2013-09-18 01:15:47,312 Stage-1 map = 100%, reduce = 72%, Cumulative CPU 1558.8 sec
2013-09-18 01:15:48,318 Stage-1 map = 100%, reduce = 73%, Cumulative CPU 1558.8 sec
2013-09-18 01:15:49,324 Stage-1 map = 100%, reduce = 75%, Cumulative CPU 1558.8 sec
2013-09-18 01:15:50,329 Stage-1 map = 100%, reduce = 75%, Cumulative CPU 1558.8 sec
2013-09-18 01:15:51,335 Stage-1 map = 100%, reduce = 75%, Cumulative CPU 1558.8 sec
2013-09-18 01:15:52,341 Stage-1 map = 100%, reduce = 76%, Cumulative CPU 1558.8 sec
2013-09-18 01:15:53,347 Stage-1 map = 100%, reduce = 76%, Cumulative CPU 1558.8 sec
2013-09-18 01:15:54,353 Stage-1 map = 100%, reduce = 76%, Cumulative CPU 1558.8 sec
2013-09-18 01:15:57,177 Stage-1 map = 100%, reduce = 78%, Cumulative CPU 1558.8 sec
2013-09-18 01:15:58,183 Stage-1 map = 100%, reduce = 78%, Cumulative CPU 1558.8 sec
2013-09-18 01:15:59,188 Stage-1 map = 100%, reduce = 78%, Cumulative CPU 1558.8 sec
2013-09-18 01:16:00,499 Stage-1 map = 100%, reduce = 72%, Cumulative CPU 1536.68 sec
2013-09-18 01:16:01,505 Stage-1 map = 100%, reduce = 73%, Cumulative CPU 1536.68 sec
2013-09-18 01:16:02,511 Stage-1 map = 100%, reduce = 73%, Cumulative CPU 1536.68 sec
2013-09-18 01:16:03,517 Stage-1 map = 100%, reduce = 73%, Cumulative CPU 1536.68 sec
2013-09-18 01:16:04,524 Stage-1 map = 100%, reduce = 73%, Cumulative CPU 1536.68 sec
2013-09-18 01:16:05,551 Stage-1 map = 100%, reduce = 73%, Cumulative CPU 1536.68 sec
2013-09-18 01:16:06,557 Stage-1 map = 100%, reduce = 74%, Cumulative CPU 1536.68 sec
2013-09-18 01:16:07,562 Stage-1 map = 100%, reduce = 75%, Cumulative CPU 1536.68 sec
2013-09-18 01:16:08,567 Stage-1 map = 100%, reduce = 75%, Cumulative CPU 1536.68 sec
2013-09-18 01:16:09,573 Stage-1 map = 100%, reduce = 76%, Cumulative CPU 1536.68 sec
2013-09-18 01:16:10,583 Stage-1 map = 100%, reduce = 76%, Cumulative CPU 1536.68 sec
2013-09-18 01:16:11,588 Stage-1 map = 100%, reduce = 76%, Cumulative CPU 1536.68 sec
2013-09-18 01:16:12,594 Stage-1 map = 100%, reduce = 78%, Cumulative CPU 1536.68 sec
2013-09-18 01:16:13,600 Stage-1 map = 100%, reduce = 78%, Cumulative CPU 1536.68 sec
2013-09-18 01:16:14,606 Stage-1 map = 100%, reduce = 78%, Cumulative CPU 1536.68 sec
2013-09-18 01:16:16,677 Stage-1 map = 100%, reduce = 81%, Cumulative CPU 1536.68 sec
2013-09-18 01:16:17,682 Stage-1 map = 100%, reduce = 81%, Cumulative CPU 1536.68 sec
2013-09-18 01:16:20,073 Stage-1 map = 100%, reduce = 82%, Cumulative CPU 1536.68 sec
2013-09-18 01:16:21,079 Stage-1 map = 100%, reduce = 82%, Cumulative CPU 1536.68 sec
2013-09-18 01:16:23,600 Stage-1 map = 100%, reduce = 87%, Cumulative CPU 1536.68 sec
2013-09-18 01:16:24,606 Stage-1 map = 100%, reduce = 87%, Cumulative CPU 1536.68 sec
2013-09-18 01:16:25,614 Stage-1 map = 100%, reduce = 87%, Cumulative CPU 1536.68 sec
2013-09-18 01:16:26,620 Stage-1 map = 100%, reduce = 89%, Cumulative CPU 1536.68 sec
2013-09-18 01:16:27,625 Stage-1 map = 100%, reduce = 89%, Cumulative CPU 1536.68 sec
2013-09-18 01:16:28,631 Stage-1 map = 100%, reduce = 89%, Cumulative CPU 1536.68 sec
2013-09-18 01:16:29,637 Stage-1 map = 100%, reduce = 92%, Cumulative CPU 1536.68 sec
2013-09-18 01:16:30,642 Stage-1 map = 100%, reduce = 92%, Cumulative CPU 1536.68 sec
2013-09-18 01:16:31,648 Stage-1 map = 100%, reduce = 92%, Cumulative CPU 1536.68 sec
2013-09-18 01:16:32,654 Stage-1 map = 100%, reduce = 94%, Cumulative CPU 1536.68 sec
2013-09-18 01:16:33,660 Stage-1 map = 100%, reduce = 94%, Cumulative CPU 1536.68 sec
2013-09-18 01:16:34,666 Stage-1 map = 100%, reduce = 94%, Cumulative CPU 1536.68 sec
2013-09-18 01:16:35,672 Stage-1 map = 100%, reduce = 96%, Cumulative CPU 1613.31 sec
2013-09-18 01:16:37,443 Stage-1 map = 100%, reduce = 96%, Cumulative CPU 1656.62 sec
2013-09-18 01:16:38,449 Stage-1 map = 100%, reduce = 97%, Cumulative CPU 1742.64 sec
2013-09-18 01:16:39,455 Stage-1 map = 100%, reduce = 97%, Cumulative CPU 1879.43 sec
2013-09-18 01:16:40,460 Stage-1 map = 100%, reduce = 98%, Cumulative CPU 1970.65 sec
2013-09-18 01:16:41,465 Stage-1 map = 100%, reduce = 98%, Cumulative CPU 1970.65 sec
2013-09-18 01:16:43,070 Stage-1 map = 100%, reduce = 98%, Cumulative CPU 1997.98 sec
2013-09-18 01:16:44,076 Stage-1 map = 100%, reduce = 98%, Cumulative CPU 1997.98 sec
2013-09-18 01:16:45,081 Stage-1 map = 100%, reduce = 98%, Cumulative CPU 1997.98 sec
2013-09-18 01:16:46,086 Stage-1 map = 100%, reduce = 98%, Cumulative CPU 1997.98 sec
2013-09-18 01:16:47,092 Stage-1 map = 100%, reduce = 98%, Cumulative CPU 1997.98 sec
2013-09-18 01:16:50,460 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1997.98 sec
2013-09-18 01:16:51,564 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1997.98 sec
2013-09-18 01:16:52,760 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1997.98 sec
2013-09-18 01:16:53,765 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1997.98 sec
2013-09-18 01:16:54,771 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1997.98 sec
2013-09-18 01:16:55,776 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1997.98 sec
2013-09-18 01:16:56,781 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1997.98 sec
2013-09-18 01:17:00,159 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1997.98 sec
2013-09-18 01:17:01,164 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1997.98 sec
2013-09-18 01:17:02,169 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1997.98 sec
2013-09-18 01:17:03,175 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1997.98 sec
2013-09-18 01:17:04,180 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1997.98 sec
2013-09-18 01:17:05,185 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1997.98 sec
2013-09-18 01:17:06,189 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1997.98 sec
2013-09-18 01:17:07,194 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1997.98 sec
2013-09-18 01:17:08,199 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1997.98 sec
2013-09-18 01:17:09,934 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 2024.84 sec
2013-09-18 01:17:10,938 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 2024.84 sec
MapReduce Total cumulative CPU time: 33 minutes 44 seconds 840 msec
Ended Job = job_201309172235_0147
Launching Job 2 out of 2
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0148
Hadoop job information for Stage-2: number of mappers: 14; number of reducers: 1
2013-09-18 01:17:14,417 Stage-2 map = 0%, reduce = 0%
2013-09-18 01:17:27,528 Stage-2 map = 1%, reduce = 0%
2013-09-18 01:17:28,533 Stage-2 map = 2%, reduce = 0%
2013-09-18 01:17:31,543 Stage-2 map = 22%, reduce = 0%
2013-09-18 01:17:37,560 Stage-2 map = 28%, reduce = 0%
2013-09-18 01:17:40,569 Stage-2 map = 43%, reduce = 0%
2013-09-18 01:17:46,588 Stage-2 map = 53%, reduce = 0%, Cumulative CPU 446.67 sec
2013-09-18 01:17:47,592 Stage-2 map = 53%, reduce = 0%, Cumulative CPU 446.67 sec
2013-09-18 01:17:48,597 Stage-2 map = 53%, reduce = 0%, Cumulative CPU 446.67 sec
2013-09-18 01:17:49,601 Stage-2 map = 64%, reduce = 0%, Cumulative CPU 446.67 sec
2013-09-18 01:17:50,606 Stage-2 map = 64%, reduce = 0%, Cumulative CPU 446.67 sec
2013-09-18 01:17:51,611 Stage-2 map = 64%, reduce = 0%, Cumulative CPU 446.67 sec
2013-09-18 01:17:52,616 Stage-2 map = 70%, reduce = 0%, Cumulative CPU 446.67 sec
2013-09-18 01:17:53,621 Stage-2 map = 70%, reduce = 0%, Cumulative CPU 446.67 sec
2013-09-18 01:17:55,731 Stage-2 map = 78%, reduce = 0%, Cumulative CPU 446.67 sec
2013-09-18 01:17:56,736 Stage-2 map = 78%, reduce = 0%, Cumulative CPU 446.67 sec
2013-09-18 01:17:57,740 Stage-2 map = 78%, reduce = 0%, Cumulative CPU 446.67 sec
2013-09-18 01:17:58,745 Stage-2 map = 79%, reduce = 0%, Cumulative CPU 467.91 sec
2013-09-18 01:17:59,749 Stage-2 map = 79%, reduce = 0%, Cumulative CPU 467.91 sec
2013-09-18 01:18:00,757 Stage-2 map = 79%, reduce = 0%, Cumulative CPU 467.91 sec
2013-09-18 01:18:01,762 Stage-2 map = 79%, reduce = 0%, Cumulative CPU 467.91 sec
2013-09-18 01:18:02,767 Stage-2 map = 79%, reduce = 0%, Cumulative CPU 467.91 sec
2013-09-18 01:18:03,771 Stage-2 map = 85%, reduce = 0%, Cumulative CPU 547.4 sec
2013-09-18 01:18:04,776 Stage-2 map = 85%, reduce = 0%, Cumulative CPU 547.4 sec
2013-09-18 01:18:05,781 Stage-2 map = 85%, reduce = 0%, Cumulative CPU 547.4 sec
2013-09-18 01:18:06,786 Stage-2 map = 85%, reduce = 0%, Cumulative CPU 547.4 sec
2013-09-18 01:18:07,791 Stage-2 map = 85%, reduce = 0%, Cumulative CPU 547.4 sec
2013-09-18 01:18:09,462 Stage-2 map = 85%, reduce = 0%, Cumulative CPU 547.4 sec
2013-09-18 01:18:10,467 Stage-2 map = 88%, reduce = 10%, Cumulative CPU 613.28 sec
2013-09-18 01:18:11,472 Stage-2 map = 95%, reduce = 10%, Cumulative CPU 782.49 sec
2013-09-18 01:18:12,482 Stage-2 map = 100%, reduce = 10%, Cumulative CPU 886.7 sec
2013-09-18 01:18:13,487 Stage-2 map = 100%, reduce = 10%, Cumulative CPU 886.7 sec
2013-09-18 01:18:14,491 Stage-2 map = 100%, reduce = 10%, Cumulative CPU 886.7 sec
2013-09-18 01:18:15,496 Stage-2 map = 100%, reduce = 10%, Cumulative CPU 886.7 sec
2013-09-18 01:18:16,501 Stage-2 map = 100%, reduce = 10%, Cumulative CPU 886.7 sec
2013-09-18 01:18:17,505 Stage-2 map = 100%, reduce = 10%, Cumulative CPU 886.7 sec
2013-09-18 01:18:18,510 Stage-2 map = 100%, reduce = 10%, Cumulative CPU 886.7 sec
2013-09-18 01:18:19,566 Stage-2 map = 100%, reduce = 10%, Cumulative CPU 886.7 sec
2013-09-18 01:18:20,571 Stage-2 map = 100%, reduce = 14%, Cumulative CPU 886.7 sec
2013-09-18 01:18:21,574 Stage-2 map = 100%, reduce = 14%, Cumulative CPU 886.7 sec
2013-09-18 01:18:22,578 Stage-2 map = 100%, reduce = 14%, Cumulative CPU 886.7 sec
2013-09-18 01:18:23,583 Stage-2 map = 100%, reduce = 14%, Cumulative CPU 886.7 sec
2013-09-18 01:18:24,587 Stage-2 map = 100%, reduce = 14%, Cumulative CPU 886.7 sec
2013-09-18 01:18:25,592 Stage-2 map = 100%, reduce = 14%, Cumulative CPU 886.7 sec
2013-09-18 01:18:26,597 Stage-2 map = 100%, reduce = 14%, Cumulative CPU 886.7 sec
2013-09-18 01:18:27,680 Stage-2 map = 100%, reduce = 19%, Cumulative CPU 886.7 sec
2013-09-18 01:18:28,684 Stage-2 map = 100%, reduce = 19%, Cumulative CPU 886.7 sec
2013-09-18 01:18:29,689 Stage-2 map = 100%, reduce = 19%, Cumulative CPU 886.7 sec
2013-09-18 01:18:30,693 Stage-2 map = 100%, reduce = 19%, Cumulative CPU 886.7 sec
2013-09-18 01:18:31,698 Stage-2 map = 100%, reduce = 19%, Cumulative CPU 886.7 sec
2013-09-18 01:18:32,702 Stage-2 map = 100%, reduce = 19%, Cumulative CPU 886.7 sec
2013-09-18 01:18:34,644 Stage-2 map = 100%, reduce = 21%, Cumulative CPU 886.7 sec
2013-09-18 01:18:35,648 Stage-2 map = 100%, reduce = 21%, Cumulative CPU 886.7 sec
2013-09-18 01:18:36,653 Stage-2 map = 100%, reduce = 21%, Cumulative CPU 886.7 sec
2013-09-18 01:18:37,657 Stage-2 map = 100%, reduce = 24%, Cumulative CPU 886.7 sec
2013-09-18 01:18:38,662 Stage-2 map = 100%, reduce = 24%, Cumulative CPU 886.7 sec
2013-09-18 01:18:39,666 Stage-2 map = 100%, reduce = 24%, Cumulative CPU 886.7 sec
2013-09-18 01:18:40,670 Stage-2 map = 100%, reduce = 26%, Cumulative CPU 886.7 sec
2013-09-18 01:18:41,674 Stage-2 map = 100%, reduce = 26%, Cumulative CPU 886.7 sec
2013-09-18 01:18:42,679 Stage-2 map = 100%, reduce = 26%, Cumulative CPU 924.39 sec
2013-09-18 01:18:43,683 Stage-2 map = 100%, reduce = 26%, Cumulative CPU 924.39 sec
2013-09-18 01:18:44,687 Stage-2 map = 100%, reduce = 26%, Cumulative CPU 924.39 sec
2013-09-18 01:18:49,434 Stage-2 map = 100%, reduce = 29%, Cumulative CPU 924.39 sec
2013-09-18 01:18:50,438 Stage-2 map = 100%, reduce = 29%, Cumulative CPU 924.39 sec
2013-09-18 01:18:51,443 Stage-2 map = 100%, reduce = 29%, Cumulative CPU 924.39 sec
2013-09-18 01:18:52,448 Stage-2 map = 100%, reduce = 31%, Cumulative CPU 924.39 sec
2013-09-18 01:18:53,452 Stage-2 map = 100%, reduce = 31%, Cumulative CPU 924.39 sec
2013-09-18 01:18:54,457 Stage-2 map = 100%, reduce = 31%, Cumulative CPU 924.39 sec
2013-09-18 01:18:55,461 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 924.39 sec
2013-09-18 01:18:56,465 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 924.39 sec
2013-09-18 01:18:57,469 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 924.39 sec
2013-09-18 01:18:58,473 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 924.39 sec
2013-09-18 01:18:59,477 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 924.39 sec
2013-09-18 01:19:00,481 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 924.39 sec
2013-09-18 01:19:05,293 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 924.39 sec
2013-09-18 01:19:06,297 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 924.39 sec
2013-09-18 01:19:07,301 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 924.39 sec
2013-09-18 01:19:08,306 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 924.39 sec
2013-09-18 01:19:09,310 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 924.39 sec
2013-09-18 01:19:10,314 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 924.39 sec
2013-09-18 01:19:11,319 Stage-2 map = 100%, reduce = 67%, Cumulative CPU 924.39 sec
2013-09-18 01:19:12,323 Stage-2 map = 100%, reduce = 67%, Cumulative CPU 924.39 sec
2013-09-18 01:19:13,331 Stage-2 map = 100%, reduce = 67%, Cumulative CPU 924.39 sec
2013-09-18 01:19:14,336 Stage-2 map = 100%, reduce = 67%, Cumulative CPU 924.39 sec
2013-09-18 01:19:16,921 Stage-2 map = 100%, reduce = 67%, Cumulative CPU 924.39 sec
2013-09-18 01:19:17,927 Stage-2 map = 100%, reduce = 67%, Cumulative CPU 924.39 sec
2013-09-18 01:19:18,931 Stage-2 map = 100%, reduce = 67%, Cumulative CPU 924.39 sec
2013-09-18 01:19:19,935 Stage-2 map = 100%, reduce = 67%, Cumulative CPU 924.39 sec
2013-09-18 01:19:20,939 Stage-2 map = 100%, reduce = 67%, Cumulative CPU 924.39 sec
2013-09-18 01:19:21,943 Stage-2 map = 100%, reduce = 67%, Cumulative CPU 924.39 sec
2013-09-18 01:19:22,947 Stage-2 map = 100%, reduce = 67%, Cumulative CPU 924.39 sec
2013-09-18 01:19:23,951 Stage-2 map = 100%, reduce = 67%, Cumulative CPU 924.39 sec
2013-09-18 01:19:24,955 Stage-2 map = 100%, reduce = 67%, Cumulative CPU 924.39 sec
2013-09-18 01:19:25,959 Stage-2 map = 100%, reduce = 67%, Cumulative CPU 924.39 sec
2013-09-18 01:19:26,963 Stage-2 map = 100%, reduce = 67%, Cumulative CPU 924.39 sec
2013-09-18 01:19:27,966 Stage-2 map = 100%, reduce = 67%, Cumulative CPU 924.39 sec
2013-09-18 01:19:28,970 Stage-2 map = 100%, reduce = 67%, Cumulative CPU 924.39 sec
2013-09-18 01:19:29,974 Stage-2 map = 100%, reduce = 67%, Cumulative CPU 924.39 sec
2013-09-18 01:19:30,977 Stage-2 map = 100%, reduce = 67%, Cumulative CPU 924.39 sec
2013-09-18 01:19:31,981 Stage-2 map = 100%, reduce = 67%, Cumulative CPU 924.39 sec
2013-09-18 01:19:32,985 Stage-2 map = 100%, reduce = 67%, Cumulative CPU 924.39 sec
2013-09-18 01:19:33,989 Stage-2 map = 100%, reduce = 67%, Cumulative CPU 924.39 sec
2013-09-18 01:19:34,992 Stage-2 map = 100%, reduce = 67%, Cumulative CPU 924.39 sec
2013-09-18 01:19:35,996 Stage-2 map = 100%, reduce = 67%, Cumulative CPU 924.39 sec
2013-09-18 01:19:37,001 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 976.38 sec
2013-09-18 01:19:38,006 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 976.38 sec
2013-09-18 01:19:39,010 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 976.38 sec
MapReduce Total cumulative CPU time: 16 minutes 16 seconds 380 msec
Ended Job = job_201309172235_0148
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 11 Cumulative CPU: 2024.84 sec HDFS Read: 869197327 HDFS Write: 4140713341 SUCCESS
Job 1: Map: 14 Reduce: 1 Cumulative CPU: 976.38 sec HDFS Read: 4140841812 HDFS Write: 417 SUCCESS
Total MapReduce CPU Time Spent: 50 minutes 1 seconds 220 msec
OK
Time taken: 324.867 seconds, Fetched: 10 row(s)
hive> quit;
times: 3
query: SELECT WatchID, ClientIP, count(*) AS c, sum(Refresh), avg(ResolutionWidth) FROM hits_100m GROUP BY WatchID, ClientIP ORDER BY c DESC LIMIT 10;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_29643@mturlrep13_201309180119_1843662809.txt
hive> SELECT WatchID, ClientIP, count(*) AS c, sum(Refresh), avg(ResolutionWidth) FROM hits_100m GROUP BY WatchID, ClientIP ORDER BY c DESC LIMIT 10;;
Total MapReduce jobs = 2
Launching Job 1 out of 2
Number of reduce tasks not specified. Estimated from input data size: 11
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0149
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 11
2013-09-18 01:19:49,884 Stage-1 map = 0%, reduce = 0%
2013-09-18 01:20:04,981 Stage-1 map = 2%, reduce = 0%
2013-09-18 01:20:05,998 Stage-1 map = 3%, reduce = 0%
2013-09-18 01:20:08,008 Stage-1 map = 9%, reduce = 0%
2013-09-18 01:20:09,017 Stage-1 map = 14%, reduce = 0%
2013-09-18 01:20:11,035 Stage-1 map = 16%, reduce = 0%
2013-09-18 01:20:12,040 Stage-1 map = 26%, reduce = 0%
2013-09-18 01:20:13,052 Stage-1 map = 27%, reduce = 0%
2013-09-18 01:20:14,066 Stage-1 map = 28%, reduce = 0%
2013-09-18 01:20:15,083 Stage-1 map = 35%, reduce = 0%
2013-09-18 01:20:16,111 Stage-1 map = 36%, reduce = 0%
2013-09-18 01:20:17,123 Stage-1 map = 37%, reduce = 0%
2013-09-18 01:20:18,128 Stage-1 map = 45%, reduce = 0%
2013-09-18 01:20:19,164 Stage-1 map = 47%, reduce = 0%
2013-09-18 01:20:21,208 Stage-1 map = 55%, reduce = 0%
2013-09-18 01:20:22,215 Stage-1 map = 58%, reduce = 0%
2013-09-18 01:20:24,224 Stage-1 map = 65%, reduce = 0%
2013-09-18 01:20:25,234 Stage-1 map = 68%, reduce = 0%
2013-09-18 01:20:27,288 Stage-1 map = 72%, reduce = 0%, Cumulative CPU 28.16 sec
2013-09-18 01:20:28,302 Stage-1 map = 79%, reduce = 0%, Cumulative CPU 28.16 sec
2013-09-18 01:20:29,308 Stage-1 map = 79%, reduce = 0%, Cumulative CPU 28.16 sec
2013-09-18 01:20:30,314 Stage-1 map = 81%, reduce = 0%, Cumulative CPU 92.89 sec
2013-09-18 01:20:31,331 Stage-1 map = 86%, reduce = 0%, Cumulative CPU 92.89 sec
2013-09-18 01:20:32,346 Stage-1 map = 87%, reduce = 0%, Cumulative CPU 193.14 sec
2013-09-18 01:20:33,367 Stage-1 map = 88%, reduce = 0%, Cumulative CPU 259.68 sec
2013-09-18 01:20:34,384 Stage-1 map = 89%, reduce = 0%, Cumulative CPU 362.61 sec
2013-09-18 01:20:35,420 Stage-1 map = 91%, reduce = 0%, Cumulative CPU 431.76 sec
2013-09-18 01:20:36,426 Stage-1 map = 93%, reduce = 0%, Cumulative CPU 683.06 sec
2013-09-18 01:20:37,432 Stage-1 map = 95%, reduce = 0%, Cumulative CPU 832.82 sec
2013-09-18 01:20:38,438 Stage-1 map = 98%, reduce = 0%, Cumulative CPU 1057.56 sec
2013-09-18 01:20:39,444 Stage-1 map = 99%, reduce = 0%, Cumulative CPU 1253.19 sec
2013-09-18 01:20:40,455 Stage-1 map = 99%, reduce = 0%, Cumulative CPU 1253.19 sec
2013-09-18 01:20:41,462 Stage-1 map = 99%, reduce = 1%, Cumulative CPU 1253.19 sec
2013-09-18 01:20:42,468 Stage-1 map = 99%, reduce = 4%, Cumulative CPU 1297.77 sec
2013-09-18 01:20:43,475 Stage-1 map = 100%, reduce = 9%, Cumulative CPU 1357.14 sec
2013-09-18 01:20:44,481 Stage-1 map = 100%, reduce = 18%, Cumulative CPU 1357.14 sec
2013-09-18 01:20:45,487 Stage-1 map = 100%, reduce = 27%, Cumulative CPU 1357.14 sec
2013-09-18 01:20:46,493 Stage-1 map = 100%, reduce = 32%, Cumulative CPU 1357.14 sec
2013-09-18 01:20:47,499 Stage-1 map = 100%, reduce = 32%, Cumulative CPU 1357.14 sec
2013-09-18 01:20:48,505 Stage-1 map = 100%, reduce = 32%, Cumulative CPU 1357.14 sec
2013-09-18 01:20:49,534 Stage-1 map = 100%, reduce = 32%, Cumulative CPU 1357.14 sec
2013-09-18 01:20:50,540 Stage-1 map = 100%, reduce = 32%, Cumulative CPU 1357.14 sec
2013-09-18 01:20:51,546 Stage-1 map = 100%, reduce = 32%, Cumulative CPU 1357.14 sec
2013-09-18 01:20:52,552 Stage-1 map = 100%, reduce = 32%, Cumulative CPU 1357.14 sec
2013-09-18 01:20:53,962 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 1357.14 sec
2013-09-18 01:20:54,968 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 1357.14 sec
2013-09-18 01:20:55,973 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 1357.14 sec
2013-09-18 01:20:56,979 Stage-1 map = 100%, reduce = 48%, Cumulative CPU 1357.14 sec
2013-09-18 01:20:57,988 Stage-1 map = 100%, reduce = 64%, Cumulative CPU 1357.14 sec
2013-09-18 01:20:59,065 Stage-1 map = 100%, reduce = 67%, Cumulative CPU 1357.14 sec
2013-09-18 01:21:00,071 Stage-1 map = 100%, reduce = 67%, Cumulative CPU 1357.14 sec
2013-09-18 01:21:01,077 Stage-1 map = 100%, reduce = 67%, Cumulative CPU 1357.14 sec
2013-09-18 01:21:02,083 Stage-1 map = 100%, reduce = 67%, Cumulative CPU 1357.14 sec
2013-09-18 01:21:03,089 Stage-1 map = 100%, reduce = 68%, Cumulative CPU 1357.14 sec
2013-09-18 01:21:04,198 Stage-1 map = 100%, reduce = 68%, Cumulative CPU 1357.14 sec
2013-09-18 01:21:05,204 Stage-1 map = 100%, reduce = 68%, Cumulative CPU 1357.14 sec
2013-09-18 01:21:06,211 Stage-1 map = 100%, reduce = 69%, Cumulative CPU 1357.14 sec
2013-09-18 01:21:07,217 Stage-1 map = 100%, reduce = 70%, Cumulative CPU 1357.14 sec
2013-09-18 01:21:08,223 Stage-1 map = 100%, reduce = 70%, Cumulative CPU 1357.14 sec
2013-09-18 01:21:09,720 Stage-1 map = 100%, reduce = 70%, Cumulative CPU 1357.14 sec
2013-09-18 01:21:10,726 Stage-1 map = 100%, reduce = 71%, Cumulative CPU 1357.14 sec
2013-09-18 01:21:11,732 Stage-1 map = 100%, reduce = 72%, Cumulative CPU 1357.14 sec
2013-09-18 01:21:12,738 Stage-1 map = 100%, reduce = 72%, Cumulative CPU 1357.14 sec
2013-09-18 01:21:13,744 Stage-1 map = 100%, reduce = 73%, Cumulative CPU 1357.14 sec
2013-09-18 01:21:16,687 Stage-1 map = 100%, reduce = 73%, Cumulative CPU 1357.14 sec
2013-09-18 01:21:17,693 Stage-1 map = 100%, reduce = 74%, Cumulative CPU 1357.14 sec
2013-09-18 01:21:18,698 Stage-1 map = 100%, reduce = 74%, Cumulative CPU 1357.14 sec
2013-09-18 01:21:19,707 Stage-1 map = 100%, reduce = 74%, Cumulative CPU 1357.14 sec
2013-09-18 01:21:20,715 Stage-1 map = 100%, reduce = 76%, Cumulative CPU 1357.14 sec
2013-09-18 01:21:21,721 Stage-1 map = 100%, reduce = 76%, Cumulative CPU 1357.14 sec
2013-09-18 01:21:25,701 Stage-1 map = 100%, reduce = 78%, Cumulative CPU 1357.14 sec
2013-09-18 01:21:26,707 Stage-1 map = 100%, reduce = 80%, Cumulative CPU 1357.14 sec
2013-09-18 01:21:28,607 Stage-1 map = 100%, reduce = 80%, Cumulative CPU 1357.14 sec
2013-09-18 01:21:29,613 Stage-1 map = 100%, reduce = 82%, Cumulative CPU 1357.14 sec
2013-09-18 01:21:30,619 Stage-1 map = 100%, reduce = 82%, Cumulative CPU 1357.14 sec
2013-09-18 01:21:31,625 Stage-1 map = 100%, reduce = 82%, Cumulative CPU 1357.14 sec
2013-09-18 01:21:32,630 Stage-1 map = 100%, reduce = 84%, Cumulative CPU 1357.14 sec
2013-09-18 01:21:33,636 Stage-1 map = 100%, reduce = 84%, Cumulative CPU 1357.14 sec
2013-09-18 01:21:34,641 Stage-1 map = 100%, reduce = 84%, Cumulative CPU 1357.14 sec
2013-09-18 01:21:35,646 Stage-1 map = 100%, reduce = 84%, Cumulative CPU 1357.14 sec
2013-09-18 01:21:37,031 Stage-1 map = 100%, reduce = 86%, Cumulative CPU 1357.14 sec
2013-09-18 01:21:38,037 Stage-1 map = 100%, reduce = 86%, Cumulative CPU 1357.14 sec
2013-09-18 01:21:39,043 Stage-1 map = 100%, reduce = 86%, Cumulative CPU 1357.14 sec
2013-09-18 01:21:40,049 Stage-1 map = 100%, reduce = 88%, Cumulative CPU 1357.14 sec
2013-09-18 01:21:41,055 Stage-1 map = 100%, reduce = 88%, Cumulative CPU 1357.14 sec
2013-09-18 01:21:42,060 Stage-1 map = 100%, reduce = 88%, Cumulative CPU 1357.14 sec
2013-09-18 01:21:43,068 Stage-1 map = 100%, reduce = 89%, Cumulative CPU 1852.58 sec
2013-09-18 01:21:44,074 Stage-1 map = 100%, reduce = 89%, Cumulative CPU 1852.58 sec
2013-09-18 01:21:45,080 Stage-1 map = 100%, reduce = 90%, Cumulative CPU 1852.58 sec
2013-09-18 01:21:46,963 Stage-1 map = 100%, reduce = 91%, Cumulative CPU 1852.58 sec
2013-09-18 01:21:47,969 Stage-1 map = 100%, reduce = 91%, Cumulative CPU 1852.58 sec
2013-09-18 01:21:48,975 Stage-1 map = 100%, reduce = 91%, Cumulative CPU 1852.58 sec
2013-09-18 01:21:52,859 Stage-1 map = 100%, reduce = 93%, Cumulative CPU 1852.58 sec
2013-09-18 01:21:54,438 Stage-1 map = 100%, reduce = 94%, Cumulative CPU 1852.58 sec
2013-09-18 01:21:55,444 Stage-1 map = 100%, reduce = 94%, Cumulative CPU 1852.58 sec
2013-09-18 01:21:57,237 Stage-1 map = 100%, reduce = 96%, Cumulative CPU 1852.58 sec
2013-09-18 01:21:58,259 Stage-1 map = 100%, reduce = 96%, Cumulative CPU 1873.66 sec
2013-09-18 01:21:59,265 Stage-1 map = 100%, reduce = 96%, Cumulative CPU 1873.66 sec
2013-09-18 01:22:00,270 Stage-1 map = 100%, reduce = 98%, Cumulative CPU 1873.66 sec
2013-09-18 01:22:01,276 Stage-1 map = 100%, reduce = 98%, Cumulative CPU 1886.44 sec
2013-09-18 01:22:02,282 Stage-1 map = 100%, reduce = 98%, Cumulative CPU 1886.44 sec
2013-09-18 01:22:03,288 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1961.16 sec
2013-09-18 01:22:04,293 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1978.53 sec
2013-09-18 01:22:05,299 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 2014.08 sec
2013-09-18 01:22:06,305 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 2014.08 sec
2013-09-18 01:22:07,310 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 2014.08 sec
MapReduce Total cumulative CPU time: 33 minutes 34 seconds 80 msec
Ended Job = job_201309172235_0149
Launching Job 2 out of 2
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0150
Hadoop job information for Stage-2: number of mappers: 14; number of reducers: 1
2013-09-18 01:22:10,699 Stage-2 map = 0%, reduce = 0%
2013-09-18 01:22:27,319 Stage-2 map = 5%, reduce = 0%
2013-09-18 01:22:30,330 Stage-2 map = 22%, reduce = 0%
2013-09-18 01:22:33,340 Stage-2 map = 25%, reduce = 0%
2013-09-18 01:22:36,352 Stage-2 map = 31%, reduce = 0%
2013-09-18 01:22:39,362 Stage-2 map = 43%, reduce = 0%
2013-09-18 01:22:42,371 Stage-2 map = 52%, reduce = 0%
2013-09-18 01:22:45,406 Stage-2 map = 62%, reduce = 0%, Cumulative CPU 517.66 sec
2013-09-18 01:22:46,420 Stage-2 map = 62%, reduce = 0%, Cumulative CPU 517.66 sec
2013-09-18 01:22:47,425 Stage-2 map = 62%, reduce = 0%, Cumulative CPU 517.66 sec
2013-09-18 01:22:48,431 Stage-2 map = 64%, reduce = 0%, Cumulative CPU 517.66 sec
2013-09-18 01:22:49,436 Stage-2 map = 64%, reduce = 0%, Cumulative CPU 517.66 sec
2013-09-18 01:22:50,441 Stage-2 map = 64%, reduce = 0%, Cumulative CPU 517.66 sec
2013-09-18 01:22:51,446 Stage-2 map = 78%, reduce = 0%, Cumulative CPU 517.66 sec
2013-09-18 01:22:54,870 Stage-2 map = 78%, reduce = 0%, Cumulative CPU 517.66 sec
2013-09-18 01:22:55,875 Stage-2 map = 78%, reduce = 0%, Cumulative CPU 517.66 sec
2013-09-18 01:22:56,880 Stage-2 map = 78%, reduce = 0%, Cumulative CPU 517.66 sec
2013-09-18 01:22:57,885 Stage-2 map = 79%, reduce = 0%, Cumulative CPU 533.57 sec
2013-09-18 01:23:00,883 Stage-2 map = 79%, reduce = 0%, Cumulative CPU 533.57 sec
2013-09-18 01:23:01,889 Stage-2 map = 81%, reduce = 0%, Cumulative CPU 553.42 sec
2013-09-18 01:23:02,895 Stage-2 map = 81%, reduce = 0%, Cumulative CPU 553.42 sec
2013-09-18 01:23:03,900 Stage-2 map = 81%, reduce = 0%, Cumulative CPU 553.42 sec
2013-09-18 01:23:04,905 Stage-2 map = 81%, reduce = 0%, Cumulative CPU 553.42 sec
2013-09-18 01:23:06,469 Stage-2 map = 85%, reduce = 0%, Cumulative CPU 597.6 sec
2013-09-18 01:23:07,474 Stage-2 map = 85%, reduce = 0%, Cumulative CPU 597.6 sec
2013-09-18 01:23:08,479 Stage-2 map = 85%, reduce = 0%, Cumulative CPU 597.6 sec
2013-09-18 01:23:09,483 Stage-2 map = 88%, reduce = 10%, Cumulative CPU 653.66 sec
2013-09-18 01:23:10,488 Stage-2 map = 92%, reduce = 10%, Cumulative CPU 738.42 sec
2013-09-18 01:23:11,492 Stage-2 map = 98%, reduce = 10%, Cumulative CPU 857.27 sec
2013-09-18 01:23:12,884 Stage-2 map = 100%, reduce = 10%, Cumulative CPU 887.38 sec
2013-09-18 01:23:14,634 Stage-2 map = 100%, reduce = 10%, Cumulative CPU 887.38 sec
2013-09-18 01:23:15,638 Stage-2 map = 100%, reduce = 12%, Cumulative CPU 887.38 sec
2013-09-18 01:23:16,643 Stage-2 map = 100%, reduce = 12%, Cumulative CPU 887.38 sec
2013-09-18 01:23:17,648 Stage-2 map = 100%, reduce = 12%, Cumulative CPU 887.38 sec
2013-09-18 01:23:18,652 Stage-2 map = 100%, reduce = 12%, Cumulative CPU 887.38 sec
2013-09-18 01:23:19,657 Stage-2 map = 100%, reduce = 12%, Cumulative CPU 887.38 sec
2013-09-18 01:23:20,662 Stage-2 map = 100%, reduce = 12%, Cumulative CPU 887.38 sec
2013-09-18 01:23:21,667 Stage-2 map = 100%, reduce = 12%, Cumulative CPU 887.38 sec
2013-09-18 01:23:22,671 Stage-2 map = 100%, reduce = 12%, Cumulative CPU 887.38 sec
2013-09-18 01:23:23,797 Stage-2 map = 100%, reduce = 17%, Cumulative CPU 887.38 sec
2013-09-18 01:23:24,801 Stage-2 map = 100%, reduce = 17%, Cumulative CPU 887.38 sec
2013-09-18 01:23:25,806 Stage-2 map = 100%, reduce = 17%, Cumulative CPU 887.38 sec
2013-09-18 01:23:26,811 Stage-2 map = 100%, reduce = 19%, Cumulative CPU 887.38 sec
2013-09-18 01:23:27,815 Stage-2 map = 100%, reduce = 19%, Cumulative CPU 887.38 sec
2013-09-18 01:23:28,820 Stage-2 map = 100%, reduce = 19%, Cumulative CPU 887.38 sec
2013-09-18 01:23:29,825 Stage-2 map = 100%, reduce = 19%, Cumulative CPU 887.38 sec
2013-09-18 01:23:31,274 Stage-2 map = 100%, reduce = 19%, Cumulative CPU 887.38 sec
2013-09-18 01:23:32,278 Stage-2 map = 100%, reduce = 19%, Cumulative CPU 887.38 sec
2013-09-18 01:23:33,283 Stage-2 map = 100%, reduce = 19%, Cumulative CPU 887.38 sec
2013-09-18 01:23:34,287 Stage-2 map = 100%, reduce = 21%, Cumulative CPU 887.38 sec
2013-09-18 01:23:35,291 Stage-2 map = 100%, reduce = 21%, Cumulative CPU 887.38 sec
2013-09-18 01:23:36,295 Stage-2 map = 100%, reduce = 21%, Cumulative CPU 887.38 sec
2013-09-18 01:23:37,300 Stage-2 map = 100%, reduce = 24%, Cumulative CPU 887.38 sec
2013-09-18 01:23:38,305 Stage-2 map = 100%, reduce = 24%, Cumulative CPU 887.38 sec
2013-09-18 01:23:39,310 Stage-2 map = 100%, reduce = 24%, Cumulative CPU 887.38 sec
2013-09-18 01:23:41,159 Stage-2 map = 100%, reduce = 24%, Cumulative CPU 887.38 sec
2013-09-18 01:23:42,164 Stage-2 map = 100%, reduce = 24%, Cumulative CPU 887.38 sec
2013-09-18 01:23:43,170 Stage-2 map = 100%, reduce = 24%, Cumulative CPU 887.38 sec
2013-09-18 01:23:44,175 Stage-2 map = 100%, reduce = 26%, Cumulative CPU 923.46 sec
2013-09-18 01:23:47,083 Stage-2 map = 100%, reduce = 26%, Cumulative CPU 923.46 sec
2013-09-18 01:23:53,300 Stage-2 map = 100%, reduce = 29%, Cumulative CPU 923.46 sec
2013-09-18 01:23:54,304 Stage-2 map = 100%, reduce = 29%, Cumulative CPU 923.46 sec
2013-09-18 01:23:55,307 Stage-2 map = 100%, reduce = 29%, Cumulative CPU 923.46 sec
2013-09-18 01:23:56,311 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 923.46 sec
2013-09-18 01:23:57,314 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 923.46 sec
2013-09-18 01:23:58,318 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 923.46 sec
2013-09-18 01:23:59,322 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 923.46 sec
2013-09-18 01:24:00,325 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 923.46 sec
2013-09-18 01:24:01,329 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 923.46 sec
2013-09-18 01:24:02,333 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 923.46 sec
2013-09-18 01:24:03,337 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 923.46 sec
2013-09-18 01:24:08,423 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 923.46 sec
2013-09-18 01:24:09,427 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 923.46 sec
2013-09-18 01:24:10,431 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 923.46 sec
2013-09-18 01:24:11,435 Stage-2 map = 100%, reduce = 67%, Cumulative CPU 923.46 sec
2013-09-18 01:24:12,439 Stage-2 map = 100%, reduce = 67%, Cumulative CPU 923.46 sec
2013-09-18 01:24:13,442 Stage-2 map = 100%, reduce = 67%, Cumulative CPU 923.46 sec
2013-09-18 01:24:14,446 Stage-2 map = 100%, reduce = 67%, Cumulative CPU 923.46 sec
2013-09-18 01:24:15,450 Stage-2 map = 100%, reduce = 67%, Cumulative CPU 923.46 sec
2013-09-18 01:24:16,454 Stage-2 map = 100%, reduce = 67%, Cumulative CPU 923.46 sec
2013-09-18 01:24:17,458 Stage-2 map = 100%, reduce = 67%, Cumulative CPU 923.46 sec
2013-09-18 01:24:18,462 Stage-2 map = 100%, reduce = 67%, Cumulative CPU 923.46 sec
2013-09-18 01:24:19,893 Stage-2 map = 100%, reduce = 67%, Cumulative CPU 923.46 sec
2013-09-18 01:24:20,896 Stage-2 map = 100%, reduce = 67%, Cumulative CPU 923.46 sec
2013-09-18 01:24:21,900 Stage-2 map = 100%, reduce = 67%, Cumulative CPU 923.46 sec
2013-09-18 01:24:22,904 Stage-2 map = 100%, reduce = 67%, Cumulative CPU 923.46 sec
2013-09-18 01:24:23,908 Stage-2 map = 100%, reduce = 67%, Cumulative CPU 923.46 sec
2013-09-18 01:24:24,918 Stage-2 map = 100%, reduce = 67%, Cumulative CPU 923.46 sec
2013-09-18 01:24:25,922 Stage-2 map = 100%, reduce = 67%, Cumulative CPU 923.46 sec
2013-09-18 01:24:26,926 Stage-2 map = 100%, reduce = 67%, Cumulative CPU 923.46 sec
2013-09-18 01:24:27,929 Stage-2 map = 100%, reduce = 67%, Cumulative CPU 923.46 sec
2013-09-18 01:24:28,932 Stage-2 map = 100%, reduce = 67%, Cumulative CPU 923.46 sec
2013-09-18 01:24:29,936 Stage-2 map = 100%, reduce = 67%, Cumulative CPU 923.46 sec
2013-09-18 01:24:30,939 Stage-2 map = 100%, reduce = 67%, Cumulative CPU 923.46 sec
2013-09-18 01:24:31,943 Stage-2 map = 100%, reduce = 67%, Cumulative CPU 923.46 sec
2013-09-18 01:24:32,947 Stage-2 map = 100%, reduce = 67%, Cumulative CPU 923.46 sec
2013-09-18 01:24:33,952 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 975.78 sec
2013-09-18 01:24:34,957 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 975.78 sec
MapReduce Total cumulative CPU time: 16 minutes 15 seconds 780 msec
Ended Job = job_201309172235_0150
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 11 Cumulative CPU: 2014.08 sec HDFS Read: 869197327 HDFS Write: 4140713341 SUCCESS
Job 1: Map: 14 Reduce: 1 Cumulative CPU: 975.78 sec HDFS Read: 4140841812 HDFS Write: 419 SUCCESS
Total MapReduce CPU Time Spent: 49 minutes 49 seconds 860 msec
OK
Time taken: 294.034 seconds, Fetched: 10 row(s)
hive> quit;
-- то же самое, но ещё и без фильтрации.;
times: 1
query: SELECT URL, count(*) AS c FROM hits_100m GROUP BY URL ORDER BY c DESC LIMIT 10;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_7945@mturlrep13_201309180124_199584274.txt
hive> SELECT URL, count(*) AS c FROM hits_100m GROUP BY URL ORDER BY c DESC LIMIT 10;;
Total MapReduce jobs = 2
Launching Job 1 out of 2
Number of reduce tasks not specified. Estimated from input data size: 11
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0151
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 11
2013-09-18 01:24:53,603 Stage-1 map = 0%, reduce = 0%
2013-09-18 01:25:03,695 Stage-1 map = 2%, reduce = 0%
2013-09-18 01:25:04,700 Stage-1 map = 3%, reduce = 0%
2013-09-18 01:25:09,065 Stage-1 map = 17%, reduce = 0%
2013-09-18 01:25:10,073 Stage-1 map = 22%, reduce = 0%
2013-09-18 01:25:11,078 Stage-1 map = 28%, reduce = 0%
2013-09-18 01:25:12,084 Stage-1 map = 31%, reduce = 0%
2013-09-18 01:25:13,091 Stage-1 map = 34%, reduce = 0%
2013-09-18 01:25:14,104 Stage-1 map = 44%, reduce = 0%
2013-09-18 01:25:15,116 Stage-1 map = 47%, reduce = 0%
2013-09-18 01:25:16,125 Stage-1 map = 50%, reduce = 0%
2013-09-18 01:25:17,131 Stage-1 map = 57%, reduce = 0%
2013-09-18 01:25:18,139 Stage-1 map = 61%, reduce = 0%
2013-09-18 01:25:20,170 Stage-1 map = 64%, reduce = 0%
2013-09-18 01:25:21,175 Stage-1 map = 66%, reduce = 0%
2013-09-18 01:25:22,187 Stage-1 map = 67%, reduce = 0%, Cumulative CPU 17.87 sec
2013-09-18 01:25:23,203 Stage-1 map = 69%, reduce = 0%, Cumulative CPU 38.23 sec
2013-09-18 01:25:24,213 Stage-1 map = 73%, reduce = 0%, Cumulative CPU 80.85 sec
2013-09-18 01:25:25,223 Stage-1 map = 75%, reduce = 0%, Cumulative CPU 123.43 sec
2013-09-18 01:25:26,229 Stage-1 map = 78%, reduce = 0%, Cumulative CPU 191.82 sec
2013-09-18 01:25:27,236 Stage-1 map = 81%, reduce = 0%, Cumulative CPU 236.59 sec
2013-09-18 01:25:28,248 Stage-1 map = 83%, reduce = 0%, Cumulative CPU 236.59 sec
2013-09-18 01:25:29,255 Stage-1 map = 85%, reduce = 0%, Cumulative CPU 284.0 sec
2013-09-18 01:25:30,261 Stage-1 map = 88%, reduce = 0%, Cumulative CPU 284.0 sec
2013-09-18 01:25:31,273 Stage-1 map = 88%, reduce = 0%, Cumulative CPU 284.0 sec
2013-09-18 01:25:32,278 Stage-1 map = 89%, reduce = 0%, Cumulative CPU 284.0 sec
2013-09-18 01:25:33,284 Stage-1 map = 90%, reduce = 0%, Cumulative CPU 284.0 sec
2013-09-18 01:25:34,290 Stage-1 map = 92%, reduce = 0%, Cumulative CPU 428.41 sec
2013-09-18 01:25:35,759 Stage-1 map = 92%, reduce = 0%, Cumulative CPU 428.41 sec
2013-09-18 01:25:36,772 Stage-1 map = 92%, reduce = 0%, Cumulative CPU 428.41 sec
2013-09-18 01:25:37,778 Stage-1 map = 93%, reduce = 0%, Cumulative CPU 428.41 sec
2013-09-18 01:25:38,785 Stage-1 map = 94%, reduce = 0%, Cumulative CPU 565.42 sec
2013-09-18 01:25:41,597 Stage-1 map = 94%, reduce = 15%, Cumulative CPU 565.42 sec
2013-09-18 01:25:45,549 Stage-1 map = 96%, reduce = 17%, Cumulative CPU 1109.57 sec
2013-09-18 01:25:46,775 Stage-1 map = 99%, reduce = 17%, Cumulative CPU 1121.14 sec
2013-09-18 01:25:47,781 Stage-1 map = 99%, reduce = 17%, Cumulative CPU 1123.73 sec
2013-09-18 01:25:48,787 Stage-1 map = 99%, reduce = 17%, Cumulative CPU 1123.73 sec
2013-09-18 01:25:49,792 Stage-1 map = 99%, reduce = 17%, Cumulative CPU 1123.73 sec
2013-09-18 01:25:50,799 Stage-1 map = 99%, reduce = 18%, Cumulative CPU 1123.73 sec
2013-09-18 01:25:51,805 Stage-1 map = 99%, reduce = 19%, Cumulative CPU 1123.73 sec
2013-09-18 01:25:52,811 Stage-1 map = 100%, reduce = 24%, Cumulative CPU 1130.32 sec
2013-09-18 01:25:53,816 Stage-1 map = 100%, reduce = 24%, Cumulative CPU 1130.32 sec
2013-09-18 01:25:55,206 Stage-1 map = 100%, reduce = 26%, Cumulative CPU 1130.32 sec
2013-09-18 01:25:56,212 Stage-1 map = 100%, reduce = 27%, Cumulative CPU 1130.32 sec
2013-09-18 01:25:57,221 Stage-1 map = 100%, reduce = 31%, Cumulative CPU 1130.32 sec
2013-09-18 01:25:58,227 Stage-1 map = 100%, reduce = 31%, Cumulative CPU 1130.32 sec
2013-09-18 01:25:59,260 Stage-1 map = 100%, reduce = 31%, Cumulative CPU 1130.32 sec
2013-09-18 01:26:00,267 Stage-1 map = 100%, reduce = 32%, Cumulative CPU 1130.32 sec
2013-09-18 01:26:01,273 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 1130.32 sec
2013-09-18 01:26:02,279 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 1130.32 sec
2013-09-18 01:26:03,286 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 1130.32 sec
2013-09-18 01:26:04,292 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 1130.32 sec
2013-09-18 01:26:09,638 Stage-1 map = 100%, reduce = 36%, Cumulative CPU 1130.32 sec
2013-09-18 01:26:10,644 Stage-1 map = 100%, reduce = 39%, Cumulative CPU 1130.32 sec
2013-09-18 01:26:14,140 Stage-1 map = 100%, reduce = 64%, Cumulative CPU 1130.32 sec
2013-09-18 01:26:15,145 Stage-1 map = 100%, reduce = 64%, Cumulative CPU 1130.32 sec
2013-09-18 01:26:16,152 Stage-1 map = 100%, reduce = 64%, Cumulative CPU 1130.32 sec
2013-09-18 01:26:17,158 Stage-1 map = 100%, reduce = 67%, Cumulative CPU 1130.32 sec
2013-09-18 01:26:18,165 Stage-1 map = 100%, reduce = 67%, Cumulative CPU 1130.32 sec
2013-09-18 01:26:19,170 Stage-1 map = 100%, reduce = 67%, Cumulative CPU 1130.32 sec
2013-09-18 01:26:20,177 Stage-1 map = 100%, reduce = 71%, Cumulative CPU 1130.32 sec
2013-09-18 01:26:21,183 Stage-1 map = 100%, reduce = 71%, Cumulative CPU 1130.32 sec
2013-09-18 01:26:22,189 Stage-1 map = 100%, reduce = 71%, Cumulative CPU 1130.32 sec
2013-09-18 01:26:23,194 Stage-1 map = 100%, reduce = 77%, Cumulative CPU 1130.32 sec
2013-09-18 01:26:24,203 Stage-1 map = 100%, reduce = 77%, Cumulative CPU 1130.32 sec
2013-09-18 01:26:25,208 Stage-1 map = 100%, reduce = 77%, Cumulative CPU 1130.32 sec
2013-09-18 01:26:26,409 Stage-1 map = 100%, reduce = 81%, Cumulative CPU 1130.32 sec
2013-09-18 01:26:27,415 Stage-1 map = 100%, reduce = 81%, Cumulative CPU 1130.32 sec
2013-09-18 01:26:28,420 Stage-1 map = 100%, reduce = 81%, Cumulative CPU 1130.32 sec
2013-09-18 01:26:29,426 Stage-1 map = 100%, reduce = 84%, Cumulative CPU 1130.32 sec
2013-09-18 01:26:30,432 Stage-1 map = 100%, reduce = 84%, Cumulative CPU 1130.32 sec
2013-09-18 01:26:31,439 Stage-1 map = 100%, reduce = 84%, Cumulative CPU 1130.32 sec
2013-09-18 01:26:33,621 Stage-1 map = 100%, reduce = 90%, Cumulative CPU 1130.32 sec
2013-09-18 01:26:34,626 Stage-1 map = 100%, reduce = 90%, Cumulative CPU 1130.32 sec
2013-09-18 01:26:35,631 Stage-1 map = 100%, reduce = 90%, Cumulative CPU 1130.32 sec
2013-09-18 01:26:36,637 Stage-1 map = 100%, reduce = 91%, Cumulative CPU 1130.32 sec
2013-09-18 01:26:37,643 Stage-1 map = 100%, reduce = 91%, Cumulative CPU 1130.32 sec
2013-09-18 01:26:38,648 Stage-1 map = 100%, reduce = 91%, Cumulative CPU 1130.32 sec
2013-09-18 01:26:40,155 Stage-1 map = 100%, reduce = 96%, Cumulative CPU 1130.32 sec
2013-09-18 01:26:41,161 Stage-1 map = 100%, reduce = 96%, Cumulative CPU 1130.32 sec
2013-09-18 01:26:43,159 Stage-1 map = 100%, reduce = 96%, Cumulative CPU 1130.32 sec
2013-09-18 01:26:44,308 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1494.96 sec
2013-09-18 01:26:45,314 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1494.96 sec
2013-09-18 01:26:46,518 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1494.96 sec
2013-09-18 01:26:47,523 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1494.96 sec
2013-09-18 01:26:48,532 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 1494.96 sec
2013-09-18 01:26:49,538 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 1526.79 sec
2013-09-18 01:26:50,543 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 1526.79 sec
MapReduce Total cumulative CPU time: 25 minutes 26 seconds 790 msec
Ended Job = job_201309172235_0151
Launching Job 2 out of 2
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0152
Hadoop job information for Stage-2: number of mappers: 14; number of reducers: 1
2013-09-18 01:27:03,639 Stage-2 map = 0%, reduce = 0%
2013-09-18 01:27:11,681 Stage-2 map = 9%, reduce = 0%
2013-09-18 01:27:12,685 Stage-2 map = 12%, reduce = 0%
2013-09-18 01:27:15,339 Stage-2 map = 32%, reduce = 0%
2013-09-18 01:27:18,350 Stage-2 map = 59%, reduce = 0%
2013-09-18 01:27:21,361 Stage-2 map = 78%, reduce = 0%
2013-09-18 01:27:23,369 Stage-2 map = 81%, reduce = 0%, Cumulative CPU 47.02 sec
2013-09-18 01:27:24,374 Stage-2 map = 87%, reduce = 0%, Cumulative CPU 142.27 sec
2013-09-18 01:27:25,378 Stage-2 map = 90%, reduce = 0%, Cumulative CPU 190.85 sec
2013-09-18 01:27:26,383 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 350.83 sec
2013-09-18 01:27:27,388 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 350.83 sec
2013-09-18 01:27:28,392 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 350.83 sec
2013-09-18 01:27:29,400 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 350.83 sec
2013-09-18 01:27:30,405 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 350.83 sec
2013-09-18 01:27:31,411 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 350.83 sec
2013-09-18 01:27:32,416 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 350.83 sec
2013-09-18 01:27:33,421 Stage-2 map = 100%, reduce = 10%, Cumulative CPU 350.83 sec
2013-09-18 01:27:34,425 Stage-2 map = 100%, reduce = 10%, Cumulative CPU 350.83 sec
2013-09-18 01:27:35,430 Stage-2 map = 100%, reduce = 10%, Cumulative CPU 350.83 sec
2013-09-18 01:27:36,435 Stage-2 map = 100%, reduce = 12%, Cumulative CPU 350.83 sec
2013-09-18 01:27:37,440 Stage-2 map = 100%, reduce = 12%, Cumulative CPU 350.83 sec
2013-09-18 01:27:38,445 Stage-2 map = 100%, reduce = 12%, Cumulative CPU 350.83 sec
2013-09-18 01:27:39,453 Stage-2 map = 100%, reduce = 19%, Cumulative CPU 350.83 sec
2013-09-18 01:27:40,457 Stage-2 map = 100%, reduce = 19%, Cumulative CPU 350.83 sec
2013-09-18 01:27:41,461 Stage-2 map = 100%, reduce = 19%, Cumulative CPU 350.83 sec
2013-09-18 01:27:42,466 Stage-2 map = 100%, reduce = 19%, Cumulative CPU 350.83 sec
2013-09-18 01:27:43,471 Stage-2 map = 100%, reduce = 19%, Cumulative CPU 350.83 sec
2013-09-18 01:27:44,508 Stage-2 map = 100%, reduce = 19%, Cumulative CPU 368.28 sec
2013-09-18 01:27:45,513 Stage-2 map = 100%, reduce = 24%, Cumulative CPU 368.28 sec
2013-09-18 01:27:46,517 Stage-2 map = 100%, reduce = 24%, Cumulative CPU 368.28 sec
2013-09-18 01:27:47,522 Stage-2 map = 100%, reduce = 24%, Cumulative CPU 368.28 sec
2013-09-18 01:27:48,527 Stage-2 map = 100%, reduce = 29%, Cumulative CPU 368.28 sec
2013-09-18 01:27:50,078 Stage-2 map = 100%, reduce = 29%, Cumulative CPU 368.28 sec
2013-09-18 01:27:51,083 Stage-2 map = 100%, reduce = 29%, Cumulative CPU 368.28 sec
2013-09-18 01:27:52,091 Stage-2 map = 100%, reduce = 31%, Cumulative CPU 368.28 sec
2013-09-18 01:27:53,096 Stage-2 map = 100%, reduce = 31%, Cumulative CPU 368.28 sec
2013-09-18 01:27:54,100 Stage-2 map = 100%, reduce = 31%, Cumulative CPU 368.28 sec
2013-09-18 01:27:55,105 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 368.28 sec
2013-09-18 01:27:56,109 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 368.28 sec
2013-09-18 01:27:58,555 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 368.28 sec
2013-09-18 01:27:59,560 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 368.28 sec
2013-09-18 01:28:00,564 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 368.28 sec
2013-09-18 01:28:02,601 Stage-2 map = 100%, reduce = 67%, Cumulative CPU 368.28 sec
2013-09-18 01:28:03,606 Stage-2 map = 100%, reduce = 67%, Cumulative CPU 368.28 sec
2013-09-18 01:28:04,611 Stage-2 map = 100%, reduce = 67%, Cumulative CPU 368.28 sec
2013-09-18 01:28:05,616 Stage-2 map = 100%, reduce = 67%, Cumulative CPU 368.28 sec
2013-09-18 01:28:06,621 Stage-2 map = 100%, reduce = 70%, Cumulative CPU 368.28 sec
2013-09-18 01:28:07,625 Stage-2 map = 100%, reduce = 70%, Cumulative CPU 368.28 sec
2013-09-18 01:28:08,630 Stage-2 map = 100%, reduce = 70%, Cumulative CPU 368.28 sec
2013-09-18 01:28:09,635 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 368.28 sec
2013-09-18 01:28:10,639 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 368.28 sec
2013-09-18 01:28:11,644 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 368.28 sec
2013-09-18 01:28:12,648 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 368.28 sec
2013-09-18 01:28:13,653 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 368.28 sec
2013-09-18 01:28:14,657 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 368.28 sec
2013-09-18 01:28:15,662 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 368.28 sec
2013-09-18 01:28:16,666 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 368.28 sec
2013-09-18 01:28:17,671 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 368.28 sec
2013-09-18 01:28:18,675 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 368.28 sec
2013-09-18 01:28:19,679 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 368.28 sec
2013-09-18 01:28:20,684 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 368.28 sec
2013-09-18 01:28:21,688 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 368.28 sec
2013-09-18 01:28:22,692 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 368.28 sec
2013-09-18 01:28:23,696 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 368.28 sec
2013-09-18 01:28:24,700 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 368.28 sec
2013-09-18 01:28:29,545 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 368.28 sec
2013-09-18 01:28:31,146 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 368.28 sec
2013-09-18 01:28:32,150 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 368.28 sec
2013-09-18 01:28:33,154 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 368.28 sec
2013-09-18 01:28:34,158 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 368.28 sec
2013-09-18 01:28:35,163 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 400.78 sec
2013-09-18 01:28:36,169 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 400.78 sec
2013-09-18 01:28:37,173 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 400.78 sec
MapReduce Total cumulative CPU time: 6 minutes 40 seconds 780 msec
Ended Job = job_201309172235_0152
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 11 Cumulative CPU: 1526.79 sec HDFS Read: 882764316 HDFS Write: 4155848594 SUCCESS
Job 1: Map: 14 Reduce: 1 Cumulative CPU: 400.78 sec HDFS Read: 4156020610 HDFS Write: 399 SUCCESS
Total MapReduce CPU Time Spent: 32 minutes 7 seconds 570 msec
OK
Time taken: 233.937 seconds, Fetched: 10 row(s)
hive> quit;
times: 2
query: SELECT URL, count(*) AS c FROM hits_100m GROUP BY URL ORDER BY c DESC LIMIT 10;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_17325@mturlrep13_201309180128_811787741.txt
hive> SELECT URL, count(*) AS c FROM hits_100m GROUP BY URL ORDER BY c DESC LIMIT 10;;
Total MapReduce jobs = 2
Launching Job 1 out of 2
Number of reduce tasks not specified. Estimated from input data size: 11
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0153
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 11
2013-09-18 01:28:48,068 Stage-1 map = 0%, reduce = 0%
2013-09-18 01:28:58,179 Stage-1 map = 7%, reduce = 0%
2013-09-18 01:29:01,193 Stage-1 map = 12%, reduce = 0%
2013-09-18 01:29:02,199 Stage-1 map = 16%, reduce = 0%
2013-09-18 01:29:03,232 Stage-1 map = 18%, reduce = 0%
2013-09-18 01:29:04,245 Stage-1 map = 25%, reduce = 0%
2013-09-18 01:29:05,251 Stage-1 map = 34%, reduce = 0%
2013-09-18 01:29:06,256 Stage-1 map = 38%, reduce = 0%
2013-09-18 01:29:07,261 Stage-1 map = 44%, reduce = 0%
2013-09-18 01:29:08,269 Stage-1 map = 50%, reduce = 0%
2013-09-18 01:29:09,284 Stage-1 map = 56%, reduce = 0%
2013-09-18 01:29:10,289 Stage-1 map = 57%, reduce = 0%
2013-09-18 01:29:11,317 Stage-1 map = 60%, reduce = 0%
2013-09-18 01:29:12,323 Stage-1 map = 63%, reduce = 0%
2013-09-18 01:29:13,344 Stage-1 map = 64%, reduce = 0%, Cumulative CPU 17.02 sec
2013-09-18 01:29:14,380 Stage-1 map = 67%, reduce = 0%, Cumulative CPU 17.02 sec
2013-09-18 01:29:15,404 Stage-1 map = 69%, reduce = 0%, Cumulative CPU 57.53 sec
2013-09-18 01:29:16,412 Stage-1 map = 70%, reduce = 0%, Cumulative CPU 98.75 sec
2013-09-18 01:29:17,422 Stage-1 map = 72%, reduce = 0%, Cumulative CPU 119.1 sec
2013-09-18 01:29:18,433 Stage-1 map = 76%, reduce = 0%, Cumulative CPU 183.33 sec
2013-09-18 01:29:19,440 Stage-1 map = 77%, reduce = 0%, Cumulative CPU 229.01 sec
2013-09-18 01:29:20,451 Stage-1 map = 80%, reduce = 0%, Cumulative CPU 229.01 sec
2013-09-18 01:29:21,457 Stage-1 map = 82%, reduce = 0%, Cumulative CPU 229.01 sec
2013-09-18 01:29:22,468 Stage-1 map = 83%, reduce = 0%, Cumulative CPU 276.95 sec
2013-09-18 01:29:23,483 Stage-1 map = 85%, reduce = 0%, Cumulative CPU 276.95 sec
2013-09-18 01:29:24,490 Stage-1 map = 85%, reduce = 0%, Cumulative CPU 276.95 sec
2013-09-18 01:29:25,502 Stage-1 map = 91%, reduce = 0%, Cumulative CPU 362.44 sec
2013-09-18 01:29:26,510 Stage-1 map = 91%, reduce = 0%, Cumulative CPU 362.44 sec
2013-09-18 01:29:29,051 Stage-1 map = 93%, reduce = 0%, Cumulative CPU 392.77 sec
2013-09-18 01:29:35,351 Stage-1 map = 94%, reduce = 15%, Cumulative CPU 521.09 sec
2013-09-18 01:29:37,211 Stage-1 map = 98%, reduce = 15%, Cumulative CPU 907.91 sec
2013-09-18 01:29:38,217 Stage-1 map = 99%, reduce = 15%, Cumulative CPU 946.27 sec
2013-09-18 01:29:39,222 Stage-1 map = 99%, reduce = 15%, Cumulative CPU 946.27 sec
2013-09-18 01:29:40,228 Stage-1 map = 99%, reduce = 16%, Cumulative CPU 946.27 sec
2013-09-18 01:29:41,234 Stage-1 map = 99%, reduce = 17%, Cumulative CPU 946.27 sec
2013-09-18 01:29:42,239 Stage-1 map = 99%, reduce = 18%, Cumulative CPU 946.27 sec
2013-09-18 01:29:43,245 Stage-1 map = 99%, reduce = 18%, Cumulative CPU 946.27 sec
2013-09-18 01:29:44,252 Stage-1 map = 99%, reduce = 18%, Cumulative CPU 1126.33 sec
2013-09-18 01:29:45,257 Stage-1 map = 100%, reduce = 19%, Cumulative CPU 1126.92 sec
2013-09-18 01:29:46,262 Stage-1 map = 100%, reduce = 19%, Cumulative CPU 1126.92 sec
2013-09-18 01:29:47,268 Stage-1 map = 100%, reduce = 28%, Cumulative CPU 1126.92 sec
2013-09-18 01:29:48,275 Stage-1 map = 100%, reduce = 30%, Cumulative CPU 1126.92 sec
2013-09-18 01:29:49,281 Stage-1 map = 100%, reduce = 30%, Cumulative CPU 1126.92 sec
2013-09-18 01:29:50,287 Stage-1 map = 100%, reduce = 31%, Cumulative CPU 1126.92 sec
2013-09-18 01:29:51,294 Stage-1 map = 100%, reduce = 31%, Cumulative CPU 1126.92 sec
2013-09-18 01:29:54,543 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 1126.92 sec
2013-09-18 01:29:55,549 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 1126.92 sec
2013-09-18 01:29:56,556 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 1126.92 sec
2013-09-18 01:30:01,304 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 1126.92 sec
2013-09-18 01:30:02,310 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 1126.92 sec
2013-09-18 01:30:03,316 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 1126.92 sec
2013-09-18 01:30:04,324 Stage-1 map = 100%, reduce = 67%, Cumulative CPU 1126.92 sec
2013-09-18 01:30:05,852 Stage-1 map = 100%, reduce = 67%, Cumulative CPU 1126.92 sec
2013-09-18 01:30:06,858 Stage-1 map = 100%, reduce = 67%, Cumulative CPU 1126.92 sec
2013-09-18 01:30:08,923 Stage-1 map = 100%, reduce = 67%, Cumulative CPU 1126.92 sec
2013-09-18 01:30:09,929 Stage-1 map = 100%, reduce = 67%, Cumulative CPU 1126.92 sec
2013-09-18 01:30:11,956 Stage-1 map = 100%, reduce = 71%, Cumulative CPU 1126.92 sec
2013-09-18 01:30:12,962 Stage-1 map = 100%, reduce = 71%, Cumulative CPU 1126.92 sec
2013-09-18 01:30:13,969 Stage-1 map = 100%, reduce = 71%, Cumulative CPU 1126.92 sec
2013-09-18 01:30:14,974 Stage-1 map = 100%, reduce = 73%, Cumulative CPU 1126.92 sec
2013-09-18 01:30:15,978 Stage-1 map = 100%, reduce = 73%, Cumulative CPU 1126.92 sec
2013-09-18 01:30:16,984 Stage-1 map = 100%, reduce = 73%, Cumulative CPU 1126.92 sec
2013-09-18 01:30:21,891 Stage-1 map = 100%, reduce = 74%, Cumulative CPU 1126.92 sec
2013-09-18 01:30:22,897 Stage-1 map = 100%, reduce = 79%, Cumulative CPU 1126.92 sec
2013-09-18 01:30:23,902 Stage-1 map = 100%, reduce = 79%, Cumulative CPU 1126.92 sec
2013-09-18 01:30:24,908 Stage-1 map = 100%, reduce = 79%, Cumulative CPU 1126.92 sec
2013-09-18 01:30:26,217 Stage-1 map = 100%, reduce = 83%, Cumulative CPU 1126.92 sec
2013-09-18 01:30:27,223 Stage-1 map = 100%, reduce = 83%, Cumulative CPU 1126.92 sec
2013-09-18 01:30:28,229 Stage-1 map = 100%, reduce = 83%, Cumulative CPU 1126.92 sec
2013-09-18 01:30:29,235 Stage-1 map = 100%, reduce = 89%, Cumulative CPU 1126.92 sec
2013-09-18 01:30:30,240 Stage-1 map = 100%, reduce = 89%, Cumulative CPU 1126.92 sec
2013-09-18 01:30:31,246 Stage-1 map = 100%, reduce = 89%, Cumulative CPU 1126.92 sec
2013-09-18 01:30:32,252 Stage-1 map = 100%, reduce = 89%, Cumulative CPU 1126.92 sec
2013-09-18 01:30:33,258 Stage-1 map = 100%, reduce = 89%, Cumulative CPU 1126.92 sec
2013-09-18 01:30:34,263 Stage-1 map = 100%, reduce = 92%, Cumulative CPU 1126.92 sec
2013-09-18 01:30:35,269 Stage-1 map = 100%, reduce = 96%, Cumulative CPU 1126.92 sec
2013-09-18 01:30:36,274 Stage-1 map = 100%, reduce = 96%, Cumulative CPU 1126.92 sec
2013-09-18 01:30:38,613 Stage-1 map = 100%, reduce = 98%, Cumulative CPU 1126.92 sec
2013-09-18 01:30:39,619 Stage-1 map = 100%, reduce = 98%, Cumulative CPU 1126.92 sec
2013-09-18 01:30:40,625 Stage-1 map = 100%, reduce = 98%, Cumulative CPU 1126.92 sec
2013-09-18 01:30:41,630 Stage-1 map = 100%, reduce = 98%, Cumulative CPU 1126.92 sec
2013-09-18 01:30:42,636 Stage-1 map = 100%, reduce = 98%, Cumulative CPU 1126.92 sec
2013-09-18 01:30:43,641 Stage-1 map = 100%, reduce = 98%, Cumulative CPU 1126.92 sec
2013-09-18 01:30:44,647 Stage-1 map = 100%, reduce = 98%, Cumulative CPU 1126.92 sec
2013-09-18 01:30:45,651 Stage-1 map = 100%, reduce = 98%, Cumulative CPU 1126.92 sec
2013-09-18 01:30:46,656 Stage-1 map = 100%, reduce = 98%, Cumulative CPU 1126.92 sec
2013-09-18 01:30:47,661 Stage-1 map = 100%, reduce = 98%, Cumulative CPU 1126.92 sec
2013-09-18 01:30:48,666 Stage-1 map = 100%, reduce = 98%, Cumulative CPU 1126.92 sec
2013-09-18 01:30:49,671 Stage-1 map = 100%, reduce = 98%, Cumulative CPU 1126.92 sec
2013-09-18 01:30:50,676 Stage-1 map = 100%, reduce = 98%, Cumulative CPU 1126.92 sec
2013-09-18 01:30:51,681 Stage-1 map = 100%, reduce = 98%, Cumulative CPU 1126.92 sec
2013-09-18 01:30:52,685 Stage-1 map = 100%, reduce = 98%, Cumulative CPU 1126.92 sec
2013-09-18 01:30:53,691 Stage-1 map = 100%, reduce = 98%, Cumulative CPU 1126.92 sec
2013-09-18 01:30:54,698 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1511.04 sec
2013-09-18 01:30:55,703 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1511.04 sec
2013-09-18 01:30:56,708 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1511.04 sec
2013-09-18 01:30:57,714 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1511.04 sec
2013-09-18 01:30:58,719 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1511.04 sec
2013-09-18 01:30:59,724 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1511.04 sec
2013-09-18 01:31:00,730 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1511.04 sec
2013-09-18 01:31:01,739 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1511.04 sec
2013-09-18 01:31:02,744 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1511.04 sec
2013-09-18 01:31:03,750 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1511.04 sec
2013-09-18 01:31:04,755 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1511.04 sec
2013-09-18 01:31:05,759 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1511.04 sec
2013-09-18 01:31:09,986 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1512.0 sec
2013-09-18 01:31:11,626 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1513.29 sec
2013-09-18 01:31:12,632 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1513.29 sec
2013-09-18 01:31:14,618 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1513.29 sec
2013-09-18 01:31:15,624 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1513.29 sec
2013-09-18 01:31:16,629 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1515.81 sec
2013-09-18 01:31:20,387 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 1518.52 sec
2013-09-18 01:31:21,393 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 1518.52 sec
MapReduce Total cumulative CPU time: 25 minutes 18 seconds 520 msec
Ended Job = job_201309172235_0153
Launching Job 2 out of 2
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0154
Hadoop job information for Stage-2: number of mappers: 14; number of reducers: 1
2013-09-18 01:31:35,149 Stage-2 map = 0%, reduce = 0%
2013-09-18 01:31:42,177 Stage-2 map = 13%, reduce = 0%
2013-09-18 01:31:45,189 Stage-2 map = 19%, reduce = 0%
2013-09-18 01:31:46,194 Stage-2 map = 36%, reduce = 0%
2013-09-18 01:31:49,205 Stage-2 map = 73%, reduce = 0%
2013-09-18 01:31:52,217 Stage-2 map = 82%, reduce = 0%, Cumulative CPU 45.62 sec
2013-09-18 01:31:53,223 Stage-2 map = 89%, reduce = 0%, Cumulative CPU 162.4 sec
2013-09-18 01:31:54,251 Stage-2 map = 94%, reduce = 0%, Cumulative CPU 239.19 sec
2013-09-18 01:31:55,256 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 346.31 sec
2013-09-18 01:31:56,260 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 346.31 sec
2013-09-18 01:31:57,265 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 346.31 sec
2013-09-18 01:31:58,270 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 346.31 sec
2013-09-18 01:31:59,275 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 346.31 sec
2013-09-18 01:32:00,280 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 346.31 sec
2013-09-18 01:32:01,285 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 346.31 sec
2013-09-18 01:32:02,291 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 346.31 sec
2013-09-18 01:32:03,296 Stage-2 map = 100%, reduce = 10%, Cumulative CPU 346.31 sec
2013-09-18 01:32:04,301 Stage-2 map = 100%, reduce = 10%, Cumulative CPU 346.31 sec
2013-09-18 01:32:05,307 Stage-2 map = 100%, reduce = 10%, Cumulative CPU 346.31 sec
2013-09-18 01:32:06,312 Stage-2 map = 100%, reduce = 12%, Cumulative CPU 346.31 sec
2013-09-18 01:32:07,318 Stage-2 map = 100%, reduce = 12%, Cumulative CPU 346.31 sec
2013-09-18 01:32:08,323 Stage-2 map = 100%, reduce = 12%, Cumulative CPU 346.31 sec
2013-09-18 01:32:09,328 Stage-2 map = 100%, reduce = 19%, Cumulative CPU 346.31 sec
2013-09-18 01:32:10,332 Stage-2 map = 100%, reduce = 19%, Cumulative CPU 346.31 sec
2013-09-18 01:32:11,338 Stage-2 map = 100%, reduce = 19%, Cumulative CPU 346.31 sec
2013-09-18 01:32:12,343 Stage-2 map = 100%, reduce = 19%, Cumulative CPU 346.31 sec
2013-09-18 01:32:13,348 Stage-2 map = 100%, reduce = 19%, Cumulative CPU 346.31 sec
2013-09-18 01:32:14,353 Stage-2 map = 100%, reduce = 19%, Cumulative CPU 346.31 sec
2013-09-18 01:32:15,358 Stage-2 map = 100%, reduce = 26%, Cumulative CPU 346.31 sec
2013-09-18 01:32:16,363 Stage-2 map = 100%, reduce = 26%, Cumulative CPU 346.31 sec
2013-09-18 01:32:17,368 Stage-2 map = 100%, reduce = 26%, Cumulative CPU 346.31 sec
2013-09-18 01:32:18,372 Stage-2 map = 100%, reduce = 29%, Cumulative CPU 346.31 sec
2013-09-18 01:32:19,377 Stage-2 map = 100%, reduce = 29%, Cumulative CPU 346.31 sec
2013-09-18 01:32:20,382 Stage-2 map = 100%, reduce = 29%, Cumulative CPU 346.31 sec
2013-09-18 01:32:21,387 Stage-2 map = 100%, reduce = 31%, Cumulative CPU 346.31 sec
2013-09-18 01:32:23,213 Stage-2 map = 100%, reduce = 31%, Cumulative CPU 346.31 sec
2013-09-18 01:32:24,218 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 346.31 sec
2013-09-18 01:32:25,222 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 346.31 sec
2013-09-18 01:32:28,762 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 346.31 sec
2013-09-18 01:32:29,767 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 346.31 sec
2013-09-18 01:32:30,772 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 346.31 sec
2013-09-18 01:32:31,777 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 346.31 sec
2013-09-18 01:32:34,832 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 346.31 sec
2013-09-18 01:32:35,836 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 346.31 sec
2013-09-18 01:32:36,841 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 346.31 sec
2013-09-18 01:32:38,072 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 346.31 sec
2013-09-18 01:32:39,077 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 346.31 sec
2013-09-18 01:32:40,083 Stage-2 map = 100%, reduce = 68%, Cumulative CPU 346.31 sec
2013-09-18 01:32:41,087 Stage-2 map = 100%, reduce = 68%, Cumulative CPU 346.31 sec
2013-09-18 01:32:42,093 Stage-2 map = 100%, reduce = 68%, Cumulative CPU 346.31 sec
2013-09-18 01:32:44,011 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 346.31 sec
2013-09-18 01:32:45,016 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 346.31 sec
2013-09-18 01:32:46,021 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 346.31 sec
2013-09-18 01:32:47,025 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 346.31 sec
2013-09-18 01:32:48,030 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 346.31 sec
2013-09-18 01:32:49,034 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 346.31 sec
2013-09-18 01:32:50,039 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 346.31 sec
2013-09-18 01:32:51,044 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 346.31 sec
2013-09-18 01:32:52,048 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 346.31 sec
2013-09-18 01:32:53,052 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 346.31 sec
2013-09-18 01:32:54,064 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 346.31 sec
2013-09-18 01:33:02,924 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 389.68 sec
2013-09-18 01:33:03,928 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 389.68 sec
2013-09-18 01:33:06,050 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 389.68 sec
2013-09-18 01:33:07,054 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 389.68 sec
2013-09-18 01:33:08,058 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 389.68 sec
2013-09-18 01:33:09,063 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 395.95 sec
2013-09-18 01:33:10,068 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 395.95 sec
2013-09-18 01:33:11,072 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 395.95 sec
MapReduce Total cumulative CPU time: 6 minutes 35 seconds 950 msec
Ended Job = job_201309172235_0154
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 11 Cumulative CPU: 1518.52 sec HDFS Read: 882764316 HDFS Write: 4155848594 SUCCESS
Job 1: Map: 14 Reduce: 1 Cumulative CPU: 395.95 sec HDFS Read: 4156020676 HDFS Write: 399 SUCCESS
Total MapReduce CPU Time Spent: 31 minutes 54 seconds 470 msec
OK
Time taken: 272.053 seconds, Fetched: 10 row(s)
hive> quit;
times: 3
query: SELECT URL, count(*) AS c FROM hits_100m GROUP BY URL ORDER BY c DESC LIMIT 10;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_26732@mturlrep13_201309180133_998177234.txt
hive> SELECT URL, count(*) AS c FROM hits_100m GROUP BY URL ORDER BY c DESC LIMIT 10;;
Total MapReduce jobs = 2
Launching Job 1 out of 2
Number of reduce tasks not specified. Estimated from input data size: 11
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0155
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 11
2013-09-18 01:33:21,912 Stage-1 map = 0%, reduce = 0%
2013-09-18 01:33:30,987 Stage-1 map = 2%, reduce = 0%
2013-09-18 01:33:31,993 Stage-1 map = 5%, reduce = 0%
2013-09-18 01:33:32,999 Stage-1 map = 6%, reduce = 0%
2013-09-18 01:33:34,004 Stage-1 map = 7%, reduce = 0%
2013-09-18 01:33:35,010 Stage-1 map = 12%, reduce = 0%
2013-09-18 01:33:36,017 Stage-1 map = 15%, reduce = 0%
2013-09-18 01:33:37,023 Stage-1 map = 19%, reduce = 0%
2013-09-18 01:33:38,030 Stage-1 map = 27%, reduce = 0%
2013-09-18 01:33:39,035 Stage-1 map = 32%, reduce = 0%
2013-09-18 01:33:40,041 Stage-1 map = 36%, reduce = 0%
2013-09-18 01:33:41,047 Stage-1 map = 45%, reduce = 0%
2013-09-18 01:33:42,053 Stage-1 map = 49%, reduce = 0%
2013-09-18 01:33:43,073 Stage-1 map = 55%, reduce = 0%
2013-09-18 01:33:44,079 Stage-1 map = 58%, reduce = 0%
2013-09-18 01:33:45,084 Stage-1 map = 61%, reduce = 0%
2013-09-18 01:33:46,177 Stage-1 map = 63%, reduce = 0%
2013-09-18 01:33:47,185 Stage-1 map = 65%, reduce = 0%
2013-09-18 01:33:48,199 Stage-1 map = 67%, reduce = 0%, Cumulative CPU 17.75 sec
2013-09-18 01:33:49,213 Stage-1 map = 68%, reduce = 0%, Cumulative CPU 17.75 sec
2013-09-18 01:33:50,227 Stage-1 map = 70%, reduce = 0%, Cumulative CPU 58.6 sec
2013-09-18 01:33:51,238 Stage-1 map = 73%, reduce = 0%, Cumulative CPU 121.99 sec
2013-09-18 01:33:52,254 Stage-1 map = 75%, reduce = 0%, Cumulative CPU 143.37 sec
2013-09-18 01:33:53,270 Stage-1 map = 78%, reduce = 0%, Cumulative CPU 234.85 sec
2013-09-18 01:33:54,282 Stage-1 map = 81%, reduce = 0%, Cumulative CPU 282.41 sec
2013-09-18 01:33:55,289 Stage-1 map = 83%, reduce = 0%, Cumulative CPU 875.12 sec
2013-09-18 01:33:56,303 Stage-1 map = 85%, reduce = 0%, Cumulative CPU 877.62 sec
2013-09-18 01:33:57,310 Stage-1 map = 85%, reduce = 0%, Cumulative CPU 877.62 sec
2013-09-18 01:33:58,319 Stage-1 map = 90%, reduce = 0%, Cumulative CPU 882.85 sec
2013-09-18 01:33:59,327 Stage-1 map = 92%, reduce = 0%, Cumulative CPU 888.05 sec
2013-09-18 01:34:00,333 Stage-1 map = 92%, reduce = 0%, Cumulative CPU 888.05 sec
2013-09-18 01:34:03,143 Stage-1 map = 92%, reduce = 0%, Cumulative CPU 892.75 sec
2013-09-18 01:34:05,074 Stage-1 map = 92%, reduce = 0%, Cumulative CPU 892.75 sec
2013-09-18 01:34:07,380 Stage-1 map = 94%, reduce = 15%, Cumulative CPU 917.16 sec
2013-09-18 01:34:08,387 Stage-1 map = 94%, reduce = 15%, Cumulative CPU 917.16 sec
2013-09-18 01:34:09,405 Stage-1 map = 94%, reduce = 15%, Cumulative CPU 917.16 sec
2013-09-18 01:34:12,449 Stage-1 map = 97%, reduce = 16%, Cumulative CPU 991.88 sec
2013-09-18 01:34:13,455 Stage-1 map = 99%, reduce = 18%, Cumulative CPU 1042.76 sec
2013-09-18 01:34:14,461 Stage-1 map = 99%, reduce = 19%, Cumulative CPU 1042.76 sec
2013-09-18 01:34:15,467 Stage-1 map = 99%, reduce = 19%, Cumulative CPU 1042.76 sec
2013-09-18 01:34:16,473 Stage-1 map = 99%, reduce = 19%, Cumulative CPU 1085.14 sec
2013-09-18 01:34:17,478 Stage-1 map = 100%, reduce = 21%, Cumulative CPU 1095.25 sec
2013-09-18 01:34:18,484 Stage-1 map = 100%, reduce = 21%, Cumulative CPU 1095.25 sec
2013-09-18 01:34:19,489 Stage-1 map = 100%, reduce = 23%, Cumulative CPU 1095.25 sec
2013-09-18 01:34:20,495 Stage-1 map = 100%, reduce = 23%, Cumulative CPU 1095.25 sec
2013-09-18 01:34:21,500 Stage-1 map = 100%, reduce = 23%, Cumulative CPU 1095.25 sec
2013-09-18 01:34:22,513 Stage-1 map = 100%, reduce = 24%, Cumulative CPU 1095.25 sec
2013-09-18 01:34:23,989 Stage-1 map = 100%, reduce = 25%, Cumulative CPU 1095.25 sec
2013-09-18 01:34:24,995 Stage-1 map = 100%, reduce = 29%, Cumulative CPU 1095.25 sec
2013-09-18 01:34:26,001 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 1095.25 sec
2013-09-18 01:34:27,007 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 1095.25 sec
2013-09-18 01:34:28,013 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 1095.25 sec
2013-09-18 01:34:29,816 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 1095.25 sec
2013-09-18 01:34:30,821 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 1095.25 sec
2013-09-18 01:34:31,827 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 1095.25 sec
2013-09-18 01:34:32,834 Stage-1 map = 100%, reduce = 36%, Cumulative CPU 1095.25 sec
2013-09-18 01:34:33,839 Stage-1 map = 100%, reduce = 36%, Cumulative CPU 1095.25 sec
2013-09-18 01:34:34,845 Stage-1 map = 100%, reduce = 42%, Cumulative CPU 1095.25 sec
2013-09-18 01:34:38,795 Stage-1 map = 100%, reduce = 52%, Cumulative CPU 1095.25 sec
2013-09-18 01:34:40,768 Stage-1 map = 100%, reduce = 64%, Cumulative CPU 1095.25 sec
2013-09-18 01:34:43,249 Stage-1 map = 100%, reduce = 68%, Cumulative CPU 1095.25 sec
2013-09-18 01:34:44,255 Stage-1 map = 100%, reduce = 68%, Cumulative CPU 1095.25 sec
2013-09-18 01:34:45,262 Stage-1 map = 100%, reduce = 68%, Cumulative CPU 1095.25 sec
2013-09-18 01:34:46,269 Stage-1 map = 100%, reduce = 74%, Cumulative CPU 1095.25 sec
2013-09-18 01:34:47,275 Stage-1 map = 100%, reduce = 74%, Cumulative CPU 1095.25 sec
2013-09-18 01:34:48,396 Stage-1 map = 100%, reduce = 74%, Cumulative CPU 1095.25 sec
2013-09-18 01:34:49,403 Stage-1 map = 100%, reduce = 78%, Cumulative CPU 1095.25 sec
2013-09-18 01:34:50,409 Stage-1 map = 100%, reduce = 78%, Cumulative CPU 1095.25 sec
2013-09-18 01:34:51,415 Stage-1 map = 100%, reduce = 78%, Cumulative CPU 1095.25 sec
2013-09-18 01:34:52,420 Stage-1 map = 100%, reduce = 81%, Cumulative CPU 1095.25 sec
2013-09-18 01:34:53,426 Stage-1 map = 100%, reduce = 81%, Cumulative CPU 1095.25 sec
2013-09-18 01:34:54,433 Stage-1 map = 100%, reduce = 81%, Cumulative CPU 1095.25 sec
2013-09-18 01:34:55,438 Stage-1 map = 100%, reduce = 84%, Cumulative CPU 1095.25 sec
2013-09-18 01:34:57,179 Stage-1 map = 100%, reduce = 86%, Cumulative CPU 1415.82 sec
2013-09-18 01:34:58,185 Stage-1 map = 100%, reduce = 88%, Cumulative CPU 1415.82 sec
2013-09-18 01:34:59,192 Stage-1 map = 100%, reduce = 88%, Cumulative CPU 1415.82 sec
2013-09-18 01:35:00,198 Stage-1 map = 100%, reduce = 90%, Cumulative CPU 1415.82 sec
2013-09-18 01:35:01,205 Stage-1 map = 100%, reduce = 93%, Cumulative CPU 1415.82 sec
2013-09-18 01:35:02,210 Stage-1 map = 100%, reduce = 93%, Cumulative CPU 1415.82 sec
2013-09-18 01:35:03,309 Stage-1 map = 100%, reduce = 95%, Cumulative CPU 1415.82 sec
2013-09-18 01:35:04,314 Stage-1 map = 100%, reduce = 97%, Cumulative CPU 1415.82 sec
2013-09-18 01:35:05,319 Stage-1 map = 100%, reduce = 97%, Cumulative CPU 1415.82 sec
2013-09-18 01:35:06,326 Stage-1 map = 100%, reduce = 97%, Cumulative CPU 1418.06 sec
2013-09-18 01:35:07,332 Stage-1 map = 100%, reduce = 98%, Cumulative CPU 1418.06 sec
2013-09-18 01:35:08,338 Stage-1 map = 100%, reduce = 98%, Cumulative CPU 1418.06 sec
2013-09-18 01:35:09,344 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1418.06 sec
2013-09-18 01:35:11,606 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1472.49 sec
2013-09-18 01:35:12,612 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1472.49 sec
2013-09-18 01:35:16,063 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 1515.38 sec
2013-09-18 01:35:17,068 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 1515.38 sec
MapReduce Total cumulative CPU time: 25 minutes 15 seconds 380 msec
Ended Job = job_201309172235_0155
Launching Job 2 out of 2
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0156
Hadoop job information for Stage-2: number of mappers: 14; number of reducers: 1
2013-09-18 01:35:35,260 Stage-2 map = 0%, reduce = 0%
2013-09-18 01:35:46,317 Stage-2 map = 17%, reduce = 0%
2013-09-18 01:35:49,327 Stage-2 map = 53%, reduce = 0%
2013-09-18 01:35:52,338 Stage-2 map = 78%, reduce = 0%
2013-09-18 01:35:55,352 Stage-2 map = 85%, reduce = 0%, Cumulative CPU 310.42 sec
2013-09-18 01:35:56,358 Stage-2 map = 89%, reduce = 0%, Cumulative CPU 312.75 sec
2013-09-18 01:35:57,366 Stage-2 map = 97%, reduce = 0%, Cumulative CPU 320.21 sec
2013-09-18 01:35:58,371 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 325.09 sec
2013-09-18 01:35:59,376 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 325.09 sec
2013-09-18 01:36:00,380 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 325.09 sec
2013-09-18 01:36:01,386 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 325.09 sec
2013-09-18 01:36:02,391 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 325.09 sec
2013-09-18 01:36:03,397 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 325.09 sec
2013-09-18 01:36:04,402 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 325.09 sec
2013-09-18 01:36:05,407 Stage-2 map = 100%, reduce = 10%, Cumulative CPU 325.09 sec
2013-09-18 01:36:06,412 Stage-2 map = 100%, reduce = 10%, Cumulative CPU 325.09 sec
2013-09-18 01:36:07,417 Stage-2 map = 100%, reduce = 10%, Cumulative CPU 325.09 sec
2013-09-18 01:36:08,422 Stage-2 map = 100%, reduce = 12%, Cumulative CPU 325.09 sec
2013-09-18 01:36:09,427 Stage-2 map = 100%, reduce = 12%, Cumulative CPU 325.09 sec
2013-09-18 01:36:10,433 Stage-2 map = 100%, reduce = 12%, Cumulative CPU 325.09 sec
2013-09-18 01:36:11,438 Stage-2 map = 100%, reduce = 17%, Cumulative CPU 325.09 sec
2013-09-18 01:36:12,444 Stage-2 map = 100%, reduce = 17%, Cumulative CPU 325.09 sec
2013-09-18 01:36:13,449 Stage-2 map = 100%, reduce = 17%, Cumulative CPU 325.09 sec
2013-09-18 01:36:14,453 Stage-2 map = 100%, reduce = 19%, Cumulative CPU 325.09 sec
2013-09-18 01:36:15,458 Stage-2 map = 100%, reduce = 19%, Cumulative CPU 325.09 sec
2013-09-18 01:36:16,463 Stage-2 map = 100%, reduce = 19%, Cumulative CPU 325.09 sec
2013-09-18 01:36:17,468 Stage-2 map = 100%, reduce = 24%, Cumulative CPU 325.09 sec
2013-09-18 01:36:18,473 Stage-2 map = 100%, reduce = 24%, Cumulative CPU 325.09 sec
2013-09-18 01:36:19,477 Stage-2 map = 100%, reduce = 24%, Cumulative CPU 325.09 sec
2013-09-18 01:36:20,482 Stage-2 map = 100%, reduce = 29%, Cumulative CPU 325.09 sec
2013-09-18 01:36:21,486 Stage-2 map = 100%, reduce = 29%, Cumulative CPU 325.09 sec
2013-09-18 01:36:25,199 Stage-2 map = 100%, reduce = 29%, Cumulative CPU 325.09 sec
2013-09-18 01:36:26,204 Stage-2 map = 100%, reduce = 29%, Cumulative CPU 325.09 sec
2013-09-18 01:36:27,208 Stage-2 map = 100%, reduce = 29%, Cumulative CPU 325.09 sec
2013-09-18 01:36:28,213 Stage-2 map = 100%, reduce = 29%, Cumulative CPU 325.09 sec
2013-09-18 01:36:29,217 Stage-2 map = 100%, reduce = 29%, Cumulative CPU 325.09 sec
2013-09-18 01:36:32,866 Stage-2 map = 100%, reduce = 29%, Cumulative CPU 325.09 sec
2013-09-18 01:36:33,870 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 325.09 sec
2013-09-18 01:36:34,875 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 325.09 sec
2013-09-18 01:36:35,880 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 325.09 sec
2013-09-18 01:36:36,885 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 325.09 sec
2013-09-18 01:36:37,888 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 325.09 sec
2013-09-18 01:36:38,892 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 325.09 sec
2013-09-18 01:36:39,896 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 325.09 sec
2013-09-18 01:36:40,900 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 325.09 sec
2013-09-18 01:36:41,904 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 325.09 sec
2013-09-18 01:36:43,207 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 325.09 sec
2013-09-18 01:36:44,211 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 325.09 sec
2013-09-18 01:36:45,216 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 325.09 sec
2013-09-18 01:36:46,220 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 325.09 sec
2013-09-18 01:36:47,225 Stage-2 map = 100%, reduce = 70%, Cumulative CPU 325.09 sec
2013-09-18 01:36:48,377 Stage-2 map = 100%, reduce = 70%, Cumulative CPU 325.09 sec
2013-09-18 01:36:49,382 Stage-2 map = 100%, reduce = 70%, Cumulative CPU 325.09 sec
2013-09-18 01:36:50,386 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 325.09 sec
2013-09-18 01:36:51,390 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 325.09 sec
2013-09-18 01:36:52,394 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 325.09 sec
2013-09-18 01:36:53,398 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 325.09 sec
2013-09-18 01:36:54,402 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 325.09 sec
2013-09-18 01:36:55,406 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 325.09 sec
2013-09-18 01:37:04,189 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 369.99 sec
2013-09-18 01:37:05,193 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 369.99 sec
2013-09-18 01:37:06,197 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 369.99 sec
2013-09-18 01:37:07,202 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 369.99 sec
2013-09-18 01:37:08,206 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 369.99 sec
2013-09-18 01:37:09,212 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 369.99 sec
2013-09-18 01:37:10,216 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 376.06 sec
2013-09-18 01:37:11,220 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 376.06 sec
MapReduce Total cumulative CPU time: 6 minutes 16 seconds 60 msec
Ended Job = job_201309172235_0156
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 11 Cumulative CPU: 1515.38 sec HDFS Read: 882764316 HDFS Write: 4155848594 SUCCESS
Job 1: Map: 14 Reduce: 1 Cumulative CPU: 376.06 sec HDFS Read: 4156020676 HDFS Write: 399 SUCCESS
Total MapReduce CPU Time Spent: 31 minutes 31 seconds 440 msec
OK
Time taken: 238.167 seconds, Fetched: 10 row(s)
hive> quit;
-- агрегация по URL.;
times: 1
query: SELECT 1, URL, count(*) AS c FROM hits_100m GROUP BY 1, URL ORDER BY c DESC LIMIT 10;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_3752@mturlrep13_201309180137_1035052608.txt
hive> SELECT 1, URL, count(*) AS c FROM hits_100m GROUP BY 1, URL ORDER BY c DESC LIMIT 10;;
Total MapReduce jobs = 2
Launching Job 1 out of 2
Number of reduce tasks not specified. Estimated from input data size: 11
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0157
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 11
2013-09-18 01:37:29,808 Stage-1 map = 0%, reduce = 0%
2013-09-18 01:37:39,901 Stage-1 map = 5%, reduce = 0%
2013-09-18 01:37:42,922 Stage-1 map = 8%, reduce = 0%
2013-09-18 01:37:43,935 Stage-1 map = 11%, reduce = 0%
2013-09-18 01:37:44,947 Stage-1 map = 13%, reduce = 0%
2013-09-18 01:37:45,953 Stage-1 map = 18%, reduce = 0%
2013-09-18 01:37:46,968 Stage-1 map = 25%, reduce = 0%
2013-09-18 01:37:47,979 Stage-1 map = 32%, reduce = 0%
2013-09-18 01:37:48,989 Stage-1 map = 36%, reduce = 0%
2013-09-18 01:37:50,002 Stage-1 map = 43%, reduce = 0%
2013-09-18 01:37:51,007 Stage-1 map = 48%, reduce = 0%
2013-09-18 01:37:52,029 Stage-1 map = 49%, reduce = 0%
2013-09-18 01:37:53,038 Stage-1 map = 54%, reduce = 0%
2013-09-18 01:37:54,051 Stage-1 map = 58%, reduce = 0%
2013-09-18 01:37:55,056 Stage-1 map = 59%, reduce = 0%
2013-09-18 01:37:56,063 Stage-1 map = 61%, reduce = 0%
2013-09-18 01:37:57,079 Stage-1 map = 65%, reduce = 0%, Cumulative CPU 669.47 sec
2013-09-18 01:37:58,095 Stage-1 map = 66%, reduce = 0%, Cumulative CPU 671.87 sec
2013-09-18 01:37:59,110 Stage-1 map = 68%, reduce = 0%, Cumulative CPU 674.43 sec
2013-09-18 01:38:00,117 Stage-1 map = 70%, reduce = 0%, Cumulative CPU 680.69 sec
2013-09-18 01:38:01,130 Stage-1 map = 75%, reduce = 0%, Cumulative CPU 692.13 sec
2013-09-18 01:38:02,144 Stage-1 map = 76%, reduce = 0%, Cumulative CPU 697.59 sec
2013-09-18 01:38:03,151 Stage-1 map = 77%, reduce = 0%, Cumulative CPU 703.09 sec
2013-09-18 01:38:04,158 Stage-1 map = 81%, reduce = 0%, Cumulative CPU 715.71 sec
2013-09-18 01:38:05,171 Stage-1 map = 83%, reduce = 0%, Cumulative CPU 724.69 sec
2013-09-18 01:38:06,182 Stage-1 map = 83%, reduce = 0%, Cumulative CPU 733.5 sec
2013-09-18 01:38:07,188 Stage-1 map = 84%, reduce = 0%, Cumulative CPU 733.5 sec
2013-09-18 01:38:12,135 Stage-1 map = 93%, reduce = 0%, Cumulative CPU 799.0 sec
2013-09-18 01:38:15,432 Stage-1 map = 94%, reduce = 5%, Cumulative CPU 849.63 sec
2013-09-18 01:38:18,103 Stage-1 map = 94%, reduce = 16%, Cumulative CPU 849.63 sec
2013-09-18 01:38:21,194 Stage-1 map = 94%, reduce = 17%, Cumulative CPU 849.63 sec
2013-09-18 01:38:22,199 Stage-1 map = 99%, reduce = 18%, Cumulative CPU 1126.64 sec
2013-09-18 01:38:23,206 Stage-1 map = 99%, reduce = 18%, Cumulative CPU 1126.64 sec
2013-09-18 01:38:24,212 Stage-1 map = 99%, reduce = 18%, Cumulative CPU 1149.35 sec
2013-09-18 01:38:25,218 Stage-1 map = 99%, reduce = 18%, Cumulative CPU 1149.35 sec
2013-09-18 01:38:26,223 Stage-1 map = 99%, reduce = 18%, Cumulative CPU 1149.35 sec
2013-09-18 01:38:27,229 Stage-1 map = 99%, reduce = 19%, Cumulative CPU 1149.35 sec
2013-09-18 01:38:28,235 Stage-1 map = 99%, reduce = 19%, Cumulative CPU 1149.35 sec
2013-09-18 01:38:29,242 Stage-1 map = 99%, reduce = 20%, Cumulative CPU 1149.35 sec
2013-09-18 01:38:30,940 Stage-1 map = 100%, reduce = 27%, Cumulative CPU 1172.28 sec
2013-09-18 01:38:31,946 Stage-1 map = 100%, reduce = 27%, Cumulative CPU 1172.28 sec
2013-09-18 01:38:32,952 Stage-1 map = 100%, reduce = 28%, Cumulative CPU 1172.28 sec
2013-09-18 01:38:33,959 Stage-1 map = 100%, reduce = 31%, Cumulative CPU 1172.28 sec
2013-09-18 01:38:34,965 Stage-1 map = 100%, reduce = 31%, Cumulative CPU 1172.28 sec
2013-09-18 01:38:37,627 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 1172.28 sec
2013-09-18 01:38:38,638 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 1172.28 sec
2013-09-18 01:38:39,961 Stage-1 map = 100%, reduce = 36%, Cumulative CPU 1172.28 sec
2013-09-18 01:38:40,968 Stage-1 map = 100%, reduce = 36%, Cumulative CPU 1172.28 sec
2013-09-18 01:38:41,978 Stage-1 map = 100%, reduce = 36%, Cumulative CPU 1172.28 sec
2013-09-18 01:38:42,985 Stage-1 map = 100%, reduce = 49%, Cumulative CPU 1172.28 sec
2013-09-18 01:38:43,991 Stage-1 map = 100%, reduce = 55%, Cumulative CPU 1172.28 sec
2013-09-18 01:38:45,341 Stage-1 map = 100%, reduce = 61%, Cumulative CPU 1172.28 sec
2013-09-18 01:38:46,346 Stage-1 map = 100%, reduce = 67%, Cumulative CPU 1172.28 sec
2013-09-18 01:38:47,352 Stage-1 map = 100%, reduce = 67%, Cumulative CPU 1172.28 sec
2013-09-18 01:38:49,208 Stage-1 map = 100%, reduce = 68%, Cumulative CPU 1172.28 sec
2013-09-18 01:38:50,214 Stage-1 map = 100%, reduce = 68%, Cumulative CPU 1172.28 sec
2013-09-18 01:38:51,221 Stage-1 map = 100%, reduce = 68%, Cumulative CPU 1172.28 sec
2013-09-18 01:38:52,229 Stage-1 map = 100%, reduce = 68%, Cumulative CPU 1172.28 sec
2013-09-18 01:38:53,235 Stage-1 map = 100%, reduce = 69%, Cumulative CPU 1172.28 sec
2013-09-18 01:38:54,242 Stage-1 map = 100%, reduce = 75%, Cumulative CPU 1172.28 sec
2013-09-18 01:38:55,248 Stage-1 map = 100%, reduce = 75%, Cumulative CPU 1172.28 sec
2013-09-18 01:38:56,281 Stage-1 map = 100%, reduce = 75%, Cumulative CPU 1172.28 sec
2013-09-18 01:38:57,288 Stage-1 map = 100%, reduce = 75%, Cumulative CPU 1172.28 sec
2013-09-18 01:38:58,296 Stage-1 map = 100%, reduce = 76%, Cumulative CPU 1172.28 sec
2013-09-18 01:38:59,305 Stage-1 map = 100%, reduce = 80%, Cumulative CPU 1469.23 sec
2013-09-18 01:39:00,313 Stage-1 map = 100%, reduce = 80%, Cumulative CPU 1469.23 sec
2013-09-18 01:39:02,538 Stage-1 map = 100%, reduce = 84%, Cumulative CPU 1469.23 sec
2013-09-18 01:39:03,544 Stage-1 map = 100%, reduce = 84%, Cumulative CPU 1469.23 sec
2013-09-18 01:39:04,551 Stage-1 map = 100%, reduce = 84%, Cumulative CPU 1469.23 sec
2013-09-18 01:39:05,560 Stage-1 map = 100%, reduce = 86%, Cumulative CPU 1469.23 sec
2013-09-18 01:39:06,566 Stage-1 map = 100%, reduce = 86%, Cumulative CPU 1469.23 sec
2013-09-18 01:39:07,700 Stage-1 map = 100%, reduce = 86%, Cumulative CPU 1469.23 sec
2013-09-18 01:39:08,706 Stage-1 map = 100%, reduce = 92%, Cumulative CPU 1469.23 sec
2013-09-18 01:39:09,713 Stage-1 map = 100%, reduce = 92%, Cumulative CPU 1469.23 sec
2013-09-18 01:39:12,763 Stage-1 map = 100%, reduce = 92%, Cumulative CPU 1469.23 sec
2013-09-18 01:39:13,794 Stage-1 map = 100%, reduce = 95%, Cumulative CPU 1469.23 sec
2013-09-18 01:39:14,799 Stage-1 map = 100%, reduce = 97%, Cumulative CPU 1469.23 sec
2013-09-18 01:39:15,805 Stage-1 map = 100%, reduce = 97%, Cumulative CPU 1469.23 sec
2013-09-18 01:39:18,636 Stage-1 map = 100%, reduce = 97%, Cumulative CPU 1469.23 sec
2013-09-18 01:39:19,643 Stage-1 map = 100%, reduce = 97%, Cumulative CPU 1469.23 sec
2013-09-18 01:39:21,526 Stage-1 map = 100%, reduce = 97%, Cumulative CPU 1479.55 sec
2013-09-18 01:39:22,532 Stage-1 map = 100%, reduce = 97%, Cumulative CPU 1479.55 sec
2013-09-18 01:39:23,538 Stage-1 map = 100%, reduce = 98%, Cumulative CPU 1479.55 sec
2013-09-18 01:39:24,543 Stage-1 map = 100%, reduce = 98%, Cumulative CPU 1490.68 sec
2013-09-18 01:39:25,549 Stage-1 map = 100%, reduce = 98%, Cumulative CPU 1490.68 sec
2013-09-18 01:39:26,555 Stage-1 map = 100%, reduce = 98%, Cumulative CPU 1490.68 sec
2013-09-18 01:39:27,560 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1490.68 sec
2013-09-18 01:39:28,565 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1490.68 sec
2013-09-18 01:39:29,571 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1490.68 sec
2013-09-18 01:39:30,577 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1490.68 sec
2013-09-18 01:39:31,766 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1504.37 sec
2013-09-18 01:39:32,771 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1504.37 sec
2013-09-18 01:39:33,777 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1504.37 sec
2013-09-18 01:39:34,782 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1543.5 sec
2013-09-18 01:39:35,788 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1543.5 sec
2013-09-18 01:39:36,793 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1559.47 sec
2013-09-18 01:39:37,798 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1559.47 sec
2013-09-18 01:39:38,803 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1572.01 sec
2013-09-18 01:39:39,808 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1572.01 sec
2013-09-18 01:39:42,552 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1585.06 sec
2013-09-18 01:39:43,556 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1585.06 sec
2013-09-18 01:39:44,715 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 1611.38 sec
2013-09-18 01:39:45,727 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 1611.38 sec
2013-09-18 01:39:46,732 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 1611.38 sec
2013-09-18 01:39:47,738 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 1611.38 sec
2013-09-18 01:39:50,360 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 1611.38 sec
2013-09-18 01:39:51,394 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 1611.38 sec
2013-09-18 01:39:52,399 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 1611.38 sec
MapReduce Total cumulative CPU time: 26 minutes 51 seconds 380 msec
Ended Job = job_201309172235_0157
Launching Job 2 out of 2
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0158
Hadoop job information for Stage-2: number of mappers: 14; number of reducers: 1
2013-09-18 01:39:56,709 Stage-2 map = 0%, reduce = 0%
2013-09-18 01:40:04,747 Stage-2 map = 15%, reduce = 0%
2013-09-18 01:40:07,758 Stage-2 map = 38%, reduce = 0%
2013-09-18 01:40:10,768 Stage-2 map = 73%, reduce = 0%
2013-09-18 01:40:13,811 Stage-2 map = 78%, reduce = 0%
2013-09-18 01:40:14,816 Stage-2 map = 86%, reduce = 0%, Cumulative CPU 115.95 sec
2013-09-18 01:40:15,821 Stage-2 map = 91%, reduce = 0%, Cumulative CPU 189.93 sec
2013-09-18 01:40:16,826 Stage-2 map = 95%, reduce = 0%, Cumulative CPU 268.74 sec
2013-09-18 01:40:17,830 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 350.63 sec
2013-09-18 01:40:18,835 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 350.63 sec
2013-09-18 01:40:19,839 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 350.63 sec
2013-09-18 01:40:20,843 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 350.63 sec
2013-09-18 01:40:21,848 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 350.63 sec
2013-09-18 01:40:22,853 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 350.63 sec
2013-09-18 01:40:23,857 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 350.63 sec
2013-09-18 01:40:24,862 Stage-2 map = 100%, reduce = 10%, Cumulative CPU 350.63 sec
2013-09-18 01:40:25,866 Stage-2 map = 100%, reduce = 10%, Cumulative CPU 350.63 sec
2013-09-18 01:40:26,871 Stage-2 map = 100%, reduce = 10%, Cumulative CPU 350.63 sec
2013-09-18 01:40:27,875 Stage-2 map = 100%, reduce = 12%, Cumulative CPU 350.63 sec
2013-09-18 01:40:28,880 Stage-2 map = 100%, reduce = 12%, Cumulative CPU 350.63 sec
2013-09-18 01:40:29,884 Stage-2 map = 100%, reduce = 12%, Cumulative CPU 350.63 sec
2013-09-18 01:40:30,889 Stage-2 map = 100%, reduce = 19%, Cumulative CPU 350.63 sec
2013-09-18 01:40:31,893 Stage-2 map = 100%, reduce = 19%, Cumulative CPU 350.63 sec
2013-09-18 01:40:32,897 Stage-2 map = 100%, reduce = 19%, Cumulative CPU 350.63 sec
2013-09-18 01:40:33,901 Stage-2 map = 100%, reduce = 19%, Cumulative CPU 350.63 sec
2013-09-18 01:40:34,905 Stage-2 map = 100%, reduce = 21%, Cumulative CPU 350.63 sec
2013-09-18 01:40:35,909 Stage-2 map = 100%, reduce = 21%, Cumulative CPU 350.63 sec
2013-09-18 01:40:36,914 Stage-2 map = 100%, reduce = 21%, Cumulative CPU 350.63 sec
2013-09-18 01:40:37,918 Stage-2 map = 100%, reduce = 24%, Cumulative CPU 350.63 sec
2013-09-18 01:40:38,923 Stage-2 map = 100%, reduce = 24%, Cumulative CPU 350.63 sec
2013-09-18 01:40:39,927 Stage-2 map = 100%, reduce = 24%, Cumulative CPU 350.63 sec
2013-09-18 01:40:40,931 Stage-2 map = 100%, reduce = 29%, Cumulative CPU 350.63 sec
2013-09-18 01:40:41,935 Stage-2 map = 100%, reduce = 29%, Cumulative CPU 350.63 sec
2013-09-18 01:40:42,939 Stage-2 map = 100%, reduce = 29%, Cumulative CPU 350.63 sec
2013-09-18 01:40:43,943 Stage-2 map = 100%, reduce = 31%, Cumulative CPU 350.63 sec
2013-09-18 01:40:44,947 Stage-2 map = 100%, reduce = 31%, Cumulative CPU 350.63 sec
2013-09-18 01:40:45,951 Stage-2 map = 100%, reduce = 31%, Cumulative CPU 350.63 sec
2013-09-18 01:40:46,956 Stage-2 map = 100%, reduce = 31%, Cumulative CPU 350.63 sec
2013-09-18 01:40:52,079 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 350.63 sec
2013-09-18 01:40:53,087 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 350.63 sec
2013-09-18 01:40:54,091 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 350.63 sec
2013-09-18 01:40:55,095 Stage-2 map = 100%, reduce = 68%, Cumulative CPU 350.63 sec
2013-09-18 01:40:56,100 Stage-2 map = 100%, reduce = 68%, Cumulative CPU 350.63 sec
2013-09-18 01:40:57,104 Stage-2 map = 100%, reduce = 68%, Cumulative CPU 350.63 sec
2013-09-18 01:40:58,109 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 350.63 sec
2013-09-18 01:40:59,113 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 392.86 sec
2013-09-18 01:41:00,117 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 392.86 sec
2013-09-18 01:41:01,220 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 392.86 sec
2013-09-18 01:41:02,224 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 392.86 sec
2013-09-18 01:41:03,228 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 392.86 sec
2013-09-18 01:41:04,232 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 392.86 sec
2013-09-18 01:41:05,237 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 392.86 sec
2013-09-18 01:41:07,415 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 392.86 sec
2013-09-18 01:41:08,419 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 392.86 sec
2013-09-18 01:41:09,423 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 392.86 sec
2013-09-18 01:41:10,427 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 392.86 sec
2013-09-18 01:41:11,431 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 392.86 sec
2013-09-18 01:41:12,435 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 392.86 sec
2013-09-18 01:41:13,439 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 392.86 sec
2013-09-18 01:41:14,522 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 392.86 sec
2013-09-18 01:41:15,526 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 392.86 sec
2013-09-18 01:41:16,530 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 392.86 sec
2013-09-18 01:41:17,534 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 392.86 sec
2013-09-18 01:41:18,538 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 392.86 sec
2013-09-18 01:41:19,543 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 400.39 sec
2013-09-18 01:41:20,547 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 400.39 sec
MapReduce Total cumulative CPU time: 6 minutes 40 seconds 390 msec
Ended Job = job_201309172235_0158
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 11 Cumulative CPU: 1611.38 sec HDFS Read: 882764316 HDFS Write: 4177941697 SUCCESS
Job 1: Map: 14 Reduce: 1 Cumulative CPU: 400.39 sec HDFS Read: 4178102019 HDFS Write: 419 SUCCESS
Total MapReduce CPU Time Spent: 33 minutes 31 seconds 770 msec
OK
Time taken: 241.283 seconds, Fetched: 10 row(s)
hive> quit;
times: 2
query: SELECT 1, URL, count(*) AS c FROM hits_100m GROUP BY 1, URL ORDER BY c DESC LIMIT 10;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_13514@mturlrep13_201309180141_1043268364.txt
hive> SELECT 1, URL, count(*) AS c FROM hits_100m GROUP BY 1, URL ORDER BY c DESC LIMIT 10;;
Total MapReduce jobs = 2
Launching Job 1 out of 2
Number of reduce tasks not specified. Estimated from input data size: 11
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0159
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 11
2013-09-18 01:41:31,551 Stage-1 map = 0%, reduce = 0%
2013-09-18 01:41:40,616 Stage-1 map = 1%, reduce = 0%
2013-09-18 01:41:41,622 Stage-1 map = 4%, reduce = 0%
2013-09-18 01:41:42,627 Stage-1 map = 5%, reduce = 0%
2013-09-18 01:41:44,636 Stage-1 map = 13%, reduce = 0%
2013-09-18 01:41:45,642 Stage-1 map = 15%, reduce = 0%
2013-09-18 01:41:46,663 Stage-1 map = 17%, reduce = 0%
2013-09-18 01:41:47,669 Stage-1 map = 23%, reduce = 0%
2013-09-18 01:41:48,691 Stage-1 map = 32%, reduce = 0%
2013-09-18 01:41:49,697 Stage-1 map = 34%, reduce = 0%
2013-09-18 01:41:50,703 Stage-1 map = 37%, reduce = 0%
2013-09-18 01:41:51,708 Stage-1 map = 47%, reduce = 0%
2013-09-18 01:41:52,719 Stage-1 map = 52%, reduce = 0%
2013-09-18 01:41:53,724 Stage-1 map = 54%, reduce = 0%
2013-09-18 01:41:54,734 Stage-1 map = 57%, reduce = 0%
2013-09-18 01:41:55,745 Stage-1 map = 59%, reduce = 0%, Cumulative CPU 18.08 sec
2013-09-18 01:41:56,770 Stage-1 map = 60%, reduce = 0%, Cumulative CPU 18.08 sec
2013-09-18 01:41:57,780 Stage-1 map = 64%, reduce = 0%, Cumulative CPU 18.08 sec
2013-09-18 01:41:58,786 Stage-1 map = 65%, reduce = 0%, Cumulative CPU 18.08 sec
2013-09-18 01:41:59,801 Stage-1 map = 66%, reduce = 0%, Cumulative CPU 706.43 sec
2013-09-18 01:42:00,824 Stage-1 map = 70%, reduce = 0%, Cumulative CPU 708.46 sec
2013-09-18 01:42:01,830 Stage-1 map = 73%, reduce = 0%, Cumulative CPU 716.88 sec
2013-09-18 01:42:02,842 Stage-1 map = 74%, reduce = 0%, Cumulative CPU 720.42 sec
2013-09-18 01:42:03,848 Stage-1 map = 78%, reduce = 0%, Cumulative CPU 736.35 sec
2013-09-18 01:42:04,853 Stage-1 map = 82%, reduce = 0%, Cumulative CPU 752.31 sec
2013-09-18 01:42:05,863 Stage-1 map = 83%, reduce = 0%, Cumulative CPU 752.31 sec
2013-09-18 01:42:06,870 Stage-1 map = 84%, reduce = 0%, Cumulative CPU 759.4 sec
2013-09-18 01:42:07,876 Stage-1 map = 87%, reduce = 0%, Cumulative CPU 759.4 sec
2013-09-18 01:42:08,882 Stage-1 map = 87%, reduce = 0%, Cumulative CPU 759.4 sec
2013-09-18 01:42:12,723 Stage-1 map = 91%, reduce = 0%, Cumulative CPU 801.07 sec
2013-09-18 01:42:13,730 Stage-1 map = 92%, reduce = 0%, Cumulative CPU 825.56 sec
2013-09-18 01:42:18,187 Stage-1 map = 92%, reduce = 0%, Cumulative CPU 825.56 sec
2013-09-18 01:42:19,197 Stage-1 map = 93%, reduce = 13%, Cumulative CPU 825.56 sec
2013-09-18 01:42:20,202 Stage-1 map = 93%, reduce = 13%, Cumulative CPU 825.56 sec
2013-09-18 01:42:21,215 Stage-1 map = 93%, reduce = 13%, Cumulative CPU 825.56 sec
2013-09-18 01:42:22,230 Stage-1 map = 97%, reduce = 17%, Cumulative CPU 976.34 sec
2013-09-18 01:42:23,237 Stage-1 map = 97%, reduce = 17%, Cumulative CPU 994.69 sec
2013-09-18 01:42:25,056 Stage-1 map = 99%, reduce = 18%, Cumulative CPU 1066.61 sec
2013-09-18 01:42:26,062 Stage-1 map = 99%, reduce = 19%, Cumulative CPU 1066.61 sec
2013-09-18 01:42:27,068 Stage-1 map = 99%, reduce = 19%, Cumulative CPU 1088.21 sec
2013-09-18 01:42:28,074 Stage-1 map = 99%, reduce = 21%, Cumulative CPU 1088.21 sec
2013-09-18 01:42:30,283 Stage-1 map = 99%, reduce = 26%, Cumulative CPU 1088.21 sec
2013-09-18 01:42:31,294 Stage-1 map = 99%, reduce = 29%, Cumulative CPU 1131.05 sec
2013-09-18 01:42:32,300 Stage-1 map = 99%, reduce = 29%, Cumulative CPU 1131.05 sec
2013-09-18 01:42:33,318 Stage-1 map = 99%, reduce = 29%, Cumulative CPU 1131.05 sec
2013-09-18 01:42:34,323 Stage-1 map = 100%, reduce = 30%, Cumulative CPU 1155.52 sec
2013-09-18 01:42:35,330 Stage-1 map = 100%, reduce = 30%, Cumulative CPU 1155.52 sec
2013-09-18 01:42:36,336 Stage-1 map = 100%, reduce = 31%, Cumulative CPU 1155.52 sec
2013-09-18 01:42:37,343 Stage-1 map = 100%, reduce = 31%, Cumulative CPU 1155.52 sec
2013-09-18 01:42:38,350 Stage-1 map = 100%, reduce = 31%, Cumulative CPU 1155.52 sec
2013-09-18 01:42:39,356 Stage-1 map = 100%, reduce = 31%, Cumulative CPU 1155.52 sec
2013-09-18 01:42:40,361 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 1155.52 sec
2013-09-18 01:42:42,047 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 1155.52 sec
2013-09-18 01:42:43,052 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 1155.52 sec
2013-09-18 01:42:44,058 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 1155.52 sec
2013-09-18 01:42:45,064 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 1155.52 sec
2013-09-18 01:42:46,070 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 1155.52 sec
2013-09-18 01:42:48,082 Stage-1 map = 100%, reduce = 55%, Cumulative CPU 1155.52 sec
2013-09-18 01:42:51,512 Stage-1 map = 100%, reduce = 55%, Cumulative CPU 1155.52 sec
2013-09-18 01:42:52,518 Stage-1 map = 100%, reduce = 55%, Cumulative CPU 1155.52 sec
2013-09-18 01:42:53,525 Stage-1 map = 100%, reduce = 55%, Cumulative CPU 1155.52 sec
2013-09-18 01:42:56,648 Stage-1 map = 100%, reduce = 55%, Cumulative CPU 1155.52 sec
2013-09-18 01:42:57,653 Stage-1 map = 100%, reduce = 55%, Cumulative CPU 1155.52 sec
2013-09-18 01:42:58,659 Stage-1 map = 100%, reduce = 55%, Cumulative CPU 1155.52 sec
2013-09-18 01:42:59,667 Stage-1 map = 100%, reduce = 56%, Cumulative CPU 1351.1 sec
2013-09-18 01:43:00,674 Stage-1 map = 100%, reduce = 56%, Cumulative CPU 1351.1 sec
2013-09-18 01:43:01,680 Stage-1 map = 100%, reduce = 56%, Cumulative CPU 1351.1 sec
2013-09-18 01:43:02,686 Stage-1 map = 100%, reduce = 72%, Cumulative CPU 1351.1 sec
2013-09-18 01:43:03,695 Stage-1 map = 100%, reduce = 72%, Cumulative CPU 1351.1 sec
2013-09-18 01:43:04,701 Stage-1 map = 100%, reduce = 72%, Cumulative CPU 1351.1 sec
2013-09-18 01:43:05,707 Stage-1 map = 100%, reduce = 76%, Cumulative CPU 1351.1 sec
2013-09-18 01:43:06,713 Stage-1 map = 100%, reduce = 76%, Cumulative CPU 1351.1 sec
2013-09-18 01:43:08,485 Stage-1 map = 100%, reduce = 80%, Cumulative CPU 1351.1 sec
2013-09-18 01:43:09,490 Stage-1 map = 100%, reduce = 80%, Cumulative CPU 1351.1 sec
2013-09-18 01:43:10,495 Stage-1 map = 100%, reduce = 80%, Cumulative CPU 1351.1 sec
2013-09-18 01:43:11,501 Stage-1 map = 100%, reduce = 81%, Cumulative CPU 1351.1 sec
2013-09-18 01:43:12,507 Stage-1 map = 100%, reduce = 81%, Cumulative CPU 1351.1 sec
2013-09-18 01:43:13,772 Stage-1 map = 100%, reduce = 81%, Cumulative CPU 1351.1 sec
2013-09-18 01:43:14,777 Stage-1 map = 100%, reduce = 81%, Cumulative CPU 1351.1 sec
2013-09-18 01:43:15,782 Stage-1 map = 100%, reduce = 81%, Cumulative CPU 1351.1 sec
2013-09-18 01:43:16,788 Stage-1 map = 100%, reduce = 84%, Cumulative CPU 1351.1 sec
2013-09-18 01:43:17,794 Stage-1 map = 100%, reduce = 84%, Cumulative CPU 1351.1 sec
2013-09-18 01:43:18,800 Stage-1 map = 100%, reduce = 84%, Cumulative CPU 1351.1 sec
2013-09-18 01:43:19,806 Stage-1 map = 100%, reduce = 90%, Cumulative CPU 1351.1 sec
2013-09-18 01:43:21,139 Stage-1 map = 100%, reduce = 90%, Cumulative CPU 1351.1 sec
2013-09-18 01:43:22,145 Stage-1 map = 100%, reduce = 90%, Cumulative CPU 1351.1 sec
2013-09-18 01:43:23,151 Stage-1 map = 100%, reduce = 94%, Cumulative CPU 1351.1 sec
2013-09-18 01:43:26,220 Stage-1 map = 100%, reduce = 94%, Cumulative CPU 1351.1 sec
2013-09-18 01:43:30,331 Stage-1 map = 100%, reduce = 96%, Cumulative CPU 1351.1 sec
2013-09-18 01:43:31,337 Stage-1 map = 100%, reduce = 96%, Cumulative CPU 1351.1 sec
2013-09-18 01:43:32,343 Stage-1 map = 100%, reduce = 98%, Cumulative CPU 1351.1 sec
2013-09-18 01:43:33,349 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1351.1 sec
2013-09-18 01:43:34,355 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1392.71 sec
2013-09-18 01:43:35,367 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 1597.17 sec
2013-09-18 01:43:36,372 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 1597.17 sec
MapReduce Total cumulative CPU time: 26 minutes 37 seconds 170 msec
Ended Job = job_201309172235_0159
Launching Job 2 out of 2
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0160
Hadoop job information for Stage-2: number of mappers: 14; number of reducers: 1
2013-09-18 01:43:38,868 Stage-2 map = 0%, reduce = 0%
2013-09-18 01:43:49,917 Stage-2 map = 6%, reduce = 0%
2013-09-18 01:43:55,242 Stage-2 map = 16%, reduce = 0%
2013-09-18 01:43:58,254 Stage-2 map = 58%, reduce = 0%
2013-09-18 01:44:01,267 Stage-2 map = 78%, reduce = 0%, Cumulative CPU 232.96 sec
2013-09-18 01:44:02,272 Stage-2 map = 80%, reduce = 0%, Cumulative CPU 236.94 sec
2013-09-18 01:44:03,277 Stage-2 map = 85%, reduce = 0%, Cumulative CPU 254.07 sec
2013-09-18 01:44:04,282 Stage-2 map = 91%, reduce = 0%, Cumulative CPU 282.18 sec
2013-09-18 01:44:05,288 Stage-2 map = 95%, reduce = 0%, Cumulative CPU 324.72 sec
2013-09-18 01:44:06,294 Stage-2 map = 98%, reduce = 0%, Cumulative CPU 324.72 sec
2013-09-18 01:44:07,299 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 336.02 sec
2013-09-18 01:44:08,304 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 336.02 sec
2013-09-18 01:44:09,309 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 336.02 sec
2013-09-18 01:44:10,314 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 336.02 sec
2013-09-18 01:44:11,319 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 336.02 sec
2013-09-18 01:44:12,324 Stage-2 map = 100%, reduce = 10%, Cumulative CPU 336.02 sec
2013-09-18 01:44:13,329 Stage-2 map = 100%, reduce = 10%, Cumulative CPU 336.02 sec
2013-09-18 01:44:14,334 Stage-2 map = 100%, reduce = 10%, Cumulative CPU 336.02 sec
2013-09-18 01:44:15,338 Stage-2 map = 100%, reduce = 12%, Cumulative CPU 336.02 sec
2013-09-18 01:44:16,343 Stage-2 map = 100%, reduce = 12%, Cumulative CPU 336.02 sec
2013-09-18 01:44:17,347 Stage-2 map = 100%, reduce = 12%, Cumulative CPU 336.02 sec
2013-09-18 01:44:18,351 Stage-2 map = 100%, reduce = 17%, Cumulative CPU 336.02 sec
2013-09-18 01:44:19,356 Stage-2 map = 100%, reduce = 17%, Cumulative CPU 336.02 sec
2013-09-18 01:44:20,365 Stage-2 map = 100%, reduce = 17%, Cumulative CPU 336.02 sec
2013-09-18 01:44:21,370 Stage-2 map = 100%, reduce = 19%, Cumulative CPU 336.02 sec
2013-09-18 01:44:22,374 Stage-2 map = 100%, reduce = 19%, Cumulative CPU 336.02 sec
2013-09-18 01:44:23,379 Stage-2 map = 100%, reduce = 19%, Cumulative CPU 336.02 sec
2013-09-18 01:44:24,383 Stage-2 map = 100%, reduce = 24%, Cumulative CPU 336.02 sec
2013-09-18 01:44:25,388 Stage-2 map = 100%, reduce = 24%, Cumulative CPU 336.02 sec
2013-09-18 01:44:26,392 Stage-2 map = 100%, reduce = 24%, Cumulative CPU 336.02 sec
2013-09-18 01:44:27,396 Stage-2 map = 100%, reduce = 29%, Cumulative CPU 336.02 sec
2013-09-18 01:44:28,401 Stage-2 map = 100%, reduce = 29%, Cumulative CPU 336.02 sec
2013-09-18 01:44:29,405 Stage-2 map = 100%, reduce = 29%, Cumulative CPU 336.02 sec
2013-09-18 01:44:30,409 Stage-2 map = 100%, reduce = 29%, Cumulative CPU 336.02 sec
2013-09-18 01:44:31,413 Stage-2 map = 100%, reduce = 29%, Cumulative CPU 336.02 sec
2013-09-18 01:44:32,417 Stage-2 map = 100%, reduce = 29%, Cumulative CPU 336.02 sec
2013-09-18 01:44:33,421 Stage-2 map = 100%, reduce = 29%, Cumulative CPU 336.02 sec
2013-09-18 01:44:34,425 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 336.02 sec
2013-09-18 01:44:37,987 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 336.02 sec
2013-09-18 01:44:38,993 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 336.02 sec
2013-09-18 01:44:39,997 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 336.02 sec
2013-09-18 01:44:41,001 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 336.02 sec
2013-09-18 01:44:42,006 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 336.02 sec
2013-09-18 01:44:43,010 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 336.02 sec
2013-09-18 01:44:44,083 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 336.02 sec
2013-09-18 01:44:45,087 Stage-2 map = 100%, reduce = 67%, Cumulative CPU 336.02 sec
2013-09-18 01:44:46,091 Stage-2 map = 100%, reduce = 67%, Cumulative CPU 336.02 sec
2013-09-18 01:44:47,095 Stage-2 map = 100%, reduce = 67%, Cumulative CPU 336.02 sec
2013-09-18 01:44:48,099 Stage-2 map = 100%, reduce = 67%, Cumulative CPU 336.02 sec
2013-09-18 01:44:49,103 Stage-2 map = 100%, reduce = 70%, Cumulative CPU 336.02 sec
2013-09-18 01:44:50,106 Stage-2 map = 100%, reduce = 70%, Cumulative CPU 336.02 sec
2013-09-18 01:44:51,478 Stage-2 map = 100%, reduce = 70%, Cumulative CPU 336.02 sec
2013-09-18 01:44:52,482 Stage-2 map = 100%, reduce = 70%, Cumulative CPU 336.02 sec
2013-09-18 01:44:53,485 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 336.02 sec
2013-09-18 01:44:54,489 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 336.02 sec
2013-09-18 01:44:55,493 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 336.02 sec
2013-09-18 01:44:56,497 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 336.02 sec
2013-09-18 01:44:57,500 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 336.02 sec
2013-09-18 01:44:58,503 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 336.02 sec
2013-09-18 01:44:59,507 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 381.91 sec
2013-09-18 01:45:00,511 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 381.91 sec
2013-09-18 01:45:01,514 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 381.91 sec
2013-09-18 01:45:02,518 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 381.91 sec
2013-09-18 01:45:03,522 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 381.91 sec
2013-09-18 01:45:04,526 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 381.91 sec
2013-09-18 01:45:05,530 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 381.91 sec
2013-09-18 01:45:06,534 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 381.91 sec
2013-09-18 01:45:07,538 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 381.91 sec
2013-09-18 01:45:08,543 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 386.71 sec
2013-09-18 01:45:09,548 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 386.71 sec
2013-09-18 01:45:10,553 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 386.71 sec
MapReduce Total cumulative CPU time: 6 minutes 26 seconds 710 msec
Ended Job = job_201309172235_0160
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 11 Cumulative CPU: 1597.17 sec HDFS Read: 882764316 HDFS Write: 4177941697 SUCCESS
Job 1: Map: 14 Reduce: 1 Cumulative CPU: 386.71 sec HDFS Read: 4178102019 HDFS Write: 419 SUCCESS
Total MapReduce CPU Time Spent: 33 minutes 3 seconds 880 msec
OK
Time taken: 228.198 seconds, Fetched: 10 row(s)
hive> quit;
times: 3
query: SELECT 1, URL, count(*) AS c FROM hits_100m GROUP BY 1, URL ORDER BY c DESC LIMIT 10;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_22530@mturlrep13_201309180145_1699071486.txt
hive> SELECT 1, URL, count(*) AS c FROM hits_100m GROUP BY 1, URL ORDER BY c DESC LIMIT 10;;
Total MapReduce jobs = 2
Launching Job 1 out of 2
Number of reduce tasks not specified. Estimated from input data size: 11
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0161
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 11
2013-09-18 01:45:20,932 Stage-1 map = 0%, reduce = 0%
2013-09-18 01:45:30,999 Stage-1 map = 2%, reduce = 0%
2013-09-18 01:45:32,006 Stage-1 map = 4%, reduce = 0%
2013-09-18 01:45:35,020 Stage-1 map = 9%, reduce = 0%
2013-09-18 01:45:36,034 Stage-1 map = 16%, reduce = 0%
2013-09-18 01:45:37,039 Stage-1 map = 17%, reduce = 0%
2013-09-18 01:45:38,045 Stage-1 map = 20%, reduce = 0%
2013-09-18 01:45:39,054 Stage-1 map = 33%, reduce = 0%
2013-09-18 01:45:41,072 Stage-1 map = 36%, reduce = 0%
2013-09-18 01:45:42,085 Stage-1 map = 50%, reduce = 0%
2013-09-18 01:45:44,098 Stage-1 map = 54%, reduce = 0%
2013-09-18 01:45:45,104 Stage-1 map = 55%, reduce = 0%
2013-09-18 01:45:46,112 Stage-1 map = 60%, reduce = 0%
2013-09-18 01:45:48,123 Stage-1 map = 62%, reduce = 0%
2013-09-18 01:45:49,139 Stage-1 map = 65%, reduce = 0%, Cumulative CPU 19.16 sec
2013-09-18 01:45:50,153 Stage-1 map = 67%, reduce = 0%, Cumulative CPU 61.29 sec
2013-09-18 01:45:51,160 Stage-1 map = 70%, reduce = 0%, Cumulative CPU 82.27 sec
2013-09-18 01:45:52,178 Stage-1 map = 73%, reduce = 0%, Cumulative CPU 104.66 sec
2013-09-18 01:45:53,184 Stage-1 map = 75%, reduce = 0%, Cumulative CPU 149.39 sec
2013-09-18 01:45:54,190 Stage-1 map = 77%, reduce = 0%, Cumulative CPU 196.78 sec
2013-09-18 01:45:55,197 Stage-1 map = 80%, reduce = 0%, Cumulative CPU 221.14 sec
2013-09-18 01:45:56,210 Stage-1 map = 82%, reduce = 0%, Cumulative CPU 271.7 sec
2013-09-18 01:45:57,219 Stage-1 map = 84%, reduce = 0%, Cumulative CPU 297.77 sec
2013-09-18 01:45:58,225 Stage-1 map = 86%, reduce = 0%, Cumulative CPU 297.77 sec
2013-09-18 01:45:59,232 Stage-1 map = 88%, reduce = 0%, Cumulative CPU 326.46 sec
2013-09-18 01:46:02,398 Stage-1 map = 91%, reduce = 0%, Cumulative CPU 976.26 sec
2013-09-18 01:46:03,406 Stage-1 map = 92%, reduce = 0%, Cumulative CPU 978.66 sec
2013-09-18 01:46:07,469 Stage-1 map = 92%, reduce = 0%, Cumulative CPU 981.5 sec
2013-09-18 01:46:08,492 Stage-1 map = 93%, reduce = 9%, Cumulative CPU 985.55 sec
2013-09-18 01:46:09,498 Stage-1 map = 93%, reduce = 9%, Cumulative CPU 990.67 sec
2013-09-18 01:46:12,267 Stage-1 map = 94%, reduce = 9%, Cumulative CPU 999.32 sec
2013-09-18 01:46:13,634 Stage-1 map = 96%, reduce = 16%, Cumulative CPU 1050.69 sec
2013-09-18 01:46:14,648 Stage-1 map = 98%, reduce = 16%, Cumulative CPU 1108.43 sec
2013-09-18 01:46:15,653 Stage-1 map = 99%, reduce = 16%, Cumulative CPU 1125.98 sec
2013-09-18 01:46:16,659 Stage-1 map = 100%, reduce = 16%, Cumulative CPU 1161.21 sec
2013-09-18 01:46:17,666 Stage-1 map = 100%, reduce = 16%, Cumulative CPU 1161.21 sec
2013-09-18 01:46:18,672 Stage-1 map = 100%, reduce = 16%, Cumulative CPU 1161.21 sec
2013-09-18 01:46:20,622 Stage-1 map = 100%, reduce = 19%, Cumulative CPU 1161.21 sec
2013-09-18 01:46:21,629 Stage-1 map = 100%, reduce = 22%, Cumulative CPU 1161.21 sec
2013-09-18 01:46:22,635 Stage-1 map = 100%, reduce = 30%, Cumulative CPU 1161.21 sec
2013-09-18 01:46:23,643 Stage-1 map = 100%, reduce = 32%, Cumulative CPU 1161.21 sec
2013-09-18 01:46:24,982 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 1161.21 sec
2013-09-18 01:46:25,989 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 1161.21 sec
2013-09-18 01:46:26,996 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 1161.21 sec
2013-09-18 01:46:28,002 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 1161.21 sec
2013-09-18 01:46:30,536 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 1161.21 sec
2013-09-18 01:46:31,543 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 1161.21 sec
2013-09-18 01:46:32,549 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 1161.21 sec
2013-09-18 01:46:35,134 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 1161.21 sec
2013-09-18 01:46:36,140 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 1161.21 sec
2013-09-18 01:46:39,099 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 1161.21 sec
2013-09-18 01:46:40,105 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 1161.21 sec
2013-09-18 01:46:41,113 Stage-1 map = 100%, reduce = 55%, Cumulative CPU 1161.21 sec
2013-09-18 01:46:42,118 Stage-1 map = 100%, reduce = 55%, Cumulative CPU 1161.21 sec
2013-09-18 01:46:43,185 Stage-1 map = 100%, reduce = 70%, Cumulative CPU 1161.21 sec
2013-09-18 01:46:44,192 Stage-1 map = 100%, reduce = 70%, Cumulative CPU 1161.21 sec
2013-09-18 01:46:45,198 Stage-1 map = 100%, reduce = 70%, Cumulative CPU 1161.21 sec
2013-09-18 01:46:46,205 Stage-1 map = 100%, reduce = 73%, Cumulative CPU 1161.21 sec
2013-09-18 01:46:47,211 Stage-1 map = 100%, reduce = 73%, Cumulative CPU 1161.21 sec
2013-09-18 01:46:48,217 Stage-1 map = 100%, reduce = 73%, Cumulative CPU 1161.21 sec
2013-09-18 01:46:49,748 Stage-1 map = 100%, reduce = 76%, Cumulative CPU 1161.21 sec
2013-09-18 01:46:50,753 Stage-1 map = 100%, reduce = 76%, Cumulative CPU 1161.21 sec
2013-09-18 01:46:51,759 Stage-1 map = 100%, reduce = 76%, Cumulative CPU 1161.21 sec
2013-09-18 01:46:52,765 Stage-1 map = 100%, reduce = 79%, Cumulative CPU 1161.21 sec
2013-09-18 01:46:53,772 Stage-1 map = 100%, reduce = 79%, Cumulative CPU 1161.21 sec
2013-09-18 01:46:54,777 Stage-1 map = 100%, reduce = 79%, Cumulative CPU 1161.21 sec
2013-09-18 01:46:56,274 Stage-1 map = 100%, reduce = 83%, Cumulative CPU 1161.21 sec
2013-09-18 01:46:57,278 Stage-1 map = 100%, reduce = 83%, Cumulative CPU 1161.21 sec
2013-09-18 01:46:58,286 Stage-1 map = 100%, reduce = 83%, Cumulative CPU 1161.21 sec
2013-09-18 01:46:59,292 Stage-1 map = 100%, reduce = 84%, Cumulative CPU 1161.21 sec
2013-09-18 01:47:00,301 Stage-1 map = 100%, reduce = 84%, Cumulative CPU 1508.73 sec
2013-09-18 01:47:01,307 Stage-1 map = 100%, reduce = 84%, Cumulative CPU 1508.73 sec
2013-09-18 01:47:02,711 Stage-1 map = 100%, reduce = 88%, Cumulative CPU 1508.73 sec
2013-09-18 01:47:03,717 Stage-1 map = 100%, reduce = 88%, Cumulative CPU 1508.73 sec
2013-09-18 01:47:04,723 Stage-1 map = 100%, reduce = 88%, Cumulative CPU 1508.73 sec
2013-09-18 01:47:05,730 Stage-1 map = 100%, reduce = 94%, Cumulative CPU 1508.73 sec
2013-09-18 01:47:09,415 Stage-1 map = 100%, reduce = 94%, Cumulative CPU 1508.73 sec
2013-09-18 01:47:13,308 Stage-1 map = 100%, reduce = 97%, Cumulative CPU 1508.73 sec
2013-09-18 01:47:14,314 Stage-1 map = 100%, reduce = 97%, Cumulative CPU 1516.27 sec
2013-09-18 01:47:15,320 Stage-1 map = 100%, reduce = 97%, Cumulative CPU 1516.27 sec
2013-09-18 01:47:16,326 Stage-1 map = 100%, reduce = 97%, Cumulative CPU 1524.18 sec
2013-09-18 01:47:17,332 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1524.18 sec
2013-09-18 01:47:18,337 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1524.18 sec
2013-09-18 01:47:19,342 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1524.18 sec
2013-09-18 01:47:20,348 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1524.18 sec
2013-09-18 01:47:21,353 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1524.18 sec
2013-09-18 01:47:22,358 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1524.18 sec
2013-09-18 01:47:23,363 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1524.18 sec
2013-09-18 01:47:24,368 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1524.18 sec
2013-09-18 01:47:25,373 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1524.18 sec
2013-09-18 01:47:26,379 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1524.18 sec
2013-09-18 01:47:27,385 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1541.34 sec
2013-09-18 01:47:28,390 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1541.34 sec
2013-09-18 01:47:29,396 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1541.34 sec
2013-09-18 01:47:30,401 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1541.34 sec
2013-09-18 01:47:31,406 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1555.39 sec
2013-09-18 01:47:32,411 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1555.39 sec
2013-09-18 01:47:33,417 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1555.39 sec
2013-09-18 01:47:34,423 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1555.39 sec
2013-09-18 01:47:35,428 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1555.39 sec
2013-09-18 01:47:36,433 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1555.39 sec
2013-09-18 01:47:37,855 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1555.39 sec
2013-09-18 01:47:38,866 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1555.39 sec
2013-09-18 01:47:39,872 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1555.39 sec
2013-09-18 01:47:40,877 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1555.39 sec
2013-09-18 01:47:41,881 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1565.2 sec
2013-09-18 01:47:42,886 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1565.2 sec
2013-09-18 01:47:44,926 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1565.2 sec
2013-09-18 01:47:45,931 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1565.2 sec
2013-09-18 01:47:46,936 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1565.2 sec
2013-09-18 01:47:47,942 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 1621.8 sec
2013-09-18 01:47:51,089 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 1621.8 sec
MapReduce Total cumulative CPU time: 27 minutes 1 seconds 800 msec
Ended Job = job_201309172235_0161
Launching Job 2 out of 2
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0162
Hadoop job information for Stage-2: number of mappers: 14; number of reducers: 1
2013-09-18 01:47:56,428 Stage-2 map = 0%, reduce = 0%
2013-09-18 01:48:04,465 Stage-2 map = 7%, reduce = 0%
2013-09-18 01:48:05,469 Stage-2 map = 10%, reduce = 0%
2013-09-18 01:48:07,477 Stage-2 map = 21%, reduce = 0%
2013-09-18 01:48:08,482 Stage-2 map = 34%, reduce = 0%
2013-09-18 01:48:10,489 Stage-2 map = 43%, reduce = 0%
2013-09-18 01:48:11,494 Stage-2 map = 70%, reduce = 0%
2013-09-18 01:48:14,505 Stage-2 map = 78%, reduce = 0%
2013-09-18 01:48:15,510 Stage-2 map = 89%, reduce = 0%, Cumulative CPU 167.69 sec
2013-09-18 01:48:16,516 Stage-2 map = 89%, reduce = 0%, Cumulative CPU 167.69 sec
2013-09-18 01:48:17,535 Stage-2 map = 94%, reduce = 0%, Cumulative CPU 246.5 sec
2013-09-18 01:48:18,541 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 356.26 sec
2013-09-18 01:48:19,545 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 356.26 sec
2013-09-18 01:48:20,551 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 356.26 sec
2013-09-18 01:48:21,555 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 356.26 sec
2013-09-18 01:48:22,560 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 356.26 sec
2013-09-18 01:48:23,565 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 356.26 sec
2013-09-18 01:48:24,570 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 356.26 sec
2013-09-18 01:48:25,574 Stage-2 map = 100%, reduce = 10%, Cumulative CPU 356.26 sec
2013-09-18 01:48:26,578 Stage-2 map = 100%, reduce = 10%, Cumulative CPU 356.26 sec
2013-09-18 01:48:27,582 Stage-2 map = 100%, reduce = 10%, Cumulative CPU 356.26 sec
2013-09-18 01:48:28,587 Stage-2 map = 100%, reduce = 12%, Cumulative CPU 356.26 sec
2013-09-18 01:48:29,591 Stage-2 map = 100%, reduce = 12%, Cumulative CPU 356.26 sec
2013-09-18 01:48:30,595 Stage-2 map = 100%, reduce = 12%, Cumulative CPU 356.26 sec
2013-09-18 01:48:31,599 Stage-2 map = 100%, reduce = 17%, Cumulative CPU 356.26 sec
2013-09-18 01:48:32,603 Stage-2 map = 100%, reduce = 17%, Cumulative CPU 356.26 sec
2013-09-18 01:48:33,606 Stage-2 map = 100%, reduce = 17%, Cumulative CPU 356.26 sec
2013-09-18 01:48:34,610 Stage-2 map = 100%, reduce = 21%, Cumulative CPU 356.26 sec
2013-09-18 01:48:35,615 Stage-2 map = 100%, reduce = 21%, Cumulative CPU 356.26 sec
2013-09-18 01:48:36,619 Stage-2 map = 100%, reduce = 21%, Cumulative CPU 356.26 sec
2013-09-18 01:48:37,624 Stage-2 map = 100%, reduce = 24%, Cumulative CPU 356.26 sec
2013-09-18 01:48:38,628 Stage-2 map = 100%, reduce = 24%, Cumulative CPU 356.26 sec
2013-09-18 01:48:39,632 Stage-2 map = 100%, reduce = 24%, Cumulative CPU 356.26 sec
2013-09-18 01:48:40,636 Stage-2 map = 100%, reduce = 29%, Cumulative CPU 356.26 sec
2013-09-18 01:48:41,641 Stage-2 map = 100%, reduce = 29%, Cumulative CPU 356.26 sec
2013-09-18 01:48:42,645 Stage-2 map = 100%, reduce = 29%, Cumulative CPU 356.26 sec
2013-09-18 01:48:43,649 Stage-2 map = 100%, reduce = 29%, Cumulative CPU 356.26 sec
2013-09-18 01:48:44,653 Stage-2 map = 100%, reduce = 29%, Cumulative CPU 356.26 sec
2013-09-18 01:48:45,658 Stage-2 map = 100%, reduce = 29%, Cumulative CPU 356.26 sec
2013-09-18 01:48:46,662 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 356.26 sec
2013-09-18 01:48:52,547 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 356.26 sec
2013-09-18 01:48:53,551 Stage-2 map = 100%, reduce = 68%, Cumulative CPU 356.26 sec
2013-09-18 01:48:54,555 Stage-2 map = 100%, reduce = 68%, Cumulative CPU 356.26 sec
2013-09-18 01:48:55,649 Stage-2 map = 100%, reduce = 68%, Cumulative CPU 356.26 sec
2013-09-18 01:48:56,653 Stage-2 map = 100%, reduce = 70%, Cumulative CPU 356.26 sec
2013-09-18 01:48:57,657 Stage-2 map = 100%, reduce = 70%, Cumulative CPU 356.26 sec
2013-09-18 01:48:58,661 Stage-2 map = 100%, reduce = 70%, Cumulative CPU 356.26 sec
2013-09-18 01:49:00,401 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 399.92 sec
2013-09-18 01:49:01,405 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 399.92 sec
2013-09-18 01:49:02,409 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 399.92 sec
2013-09-18 01:49:03,413 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 399.92 sec
2013-09-18 01:49:04,417 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 399.92 sec
2013-09-18 01:49:05,421 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 399.92 sec
2013-09-18 01:49:06,424 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 399.92 sec
2013-09-18 01:49:07,428 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 399.92 sec
2013-09-18 01:49:08,431 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 399.92 sec
2013-09-18 01:49:09,435 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 399.92 sec
2013-09-18 01:49:10,439 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 399.92 sec
2013-09-18 01:49:11,443 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 399.92 sec
2013-09-18 01:49:12,447 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 399.92 sec
2013-09-18 01:49:13,452 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 399.92 sec
2013-09-18 01:49:14,456 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 399.92 sec
2013-09-18 01:49:15,461 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 399.92 sec
2013-09-18 01:49:16,465 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 399.92 sec
2013-09-18 01:49:17,470 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 399.92 sec
2013-09-18 01:49:18,477 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 406.76 sec
2013-09-18 01:49:19,481 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 406.76 sec
MapReduce Total cumulative CPU time: 6 minutes 46 seconds 760 msec
Ended Job = job_201309172235_0162
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 11 Cumulative CPU: 1621.8 sec HDFS Read: 882764316 HDFS Write: 4177941697 SUCCESS
Job 1: Map: 14 Reduce: 1 Cumulative CPU: 406.76 sec HDFS Read: 4178102019 HDFS Write: 419 SUCCESS
Total MapReduce CPU Time Spent: 33 minutes 48 seconds 560 msec
OK
Time taken: 246.999 seconds, Fetched: 10 row(s)
hive> quit;
-- агрегация по URL и числу.;
times: 1
query: SELECT ClientIP, ClientIP - 1, ClientIP - 2, ClientIP - 3, count(*) AS c FROM hits_100m GROUP BY ClientIP, ClientIP - 1, ClientIP - 2, ClientIP - 3 ORDER BY c DESC LIMIT 10;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_31570@mturlrep13_201309180149_1050651506.txt
hive> SELECT ClientIP, ClientIP - 1, ClientIP - 2, ClientIP - 3, count(*) AS c FROM hits_100m GROUP BY ClientIP, ClientIP - 1, ClientIP - 2, ClientIP - 3 ORDER BY c DESC LIMIT 10;;
Total MapReduce jobs = 2
Launching Job 1 out of 2
Number of reduce tasks not specified. Estimated from input data size: 11
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0163
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 11
2013-09-18 01:49:38,118 Stage-1 map = 0%, reduce = 0%
2013-09-18 01:49:51,275 Stage-1 map = 2%, reduce = 0%
2013-09-18 01:49:52,282 Stage-1 map = 6%, reduce = 0%
2013-09-18 01:49:53,293 Stage-1 map = 8%, reduce = 0%
2013-09-18 01:49:54,299 Stage-1 map = 25%, reduce = 0%
2013-09-18 01:49:55,305 Stage-1 map = 31%, reduce = 0%
2013-09-18 01:49:56,314 Stage-1 map = 33%, reduce = 0%
2013-09-18 01:49:57,321 Stage-1 map = 49%, reduce = 0%
2013-09-18 01:49:58,338 Stage-1 map = 55%, reduce = 0%
2013-09-18 01:49:59,351 Stage-1 map = 60%, reduce = 0%, Cumulative CPU 13.34 sec
2013-09-18 01:50:00,378 Stage-1 map = 70%, reduce = 0%, Cumulative CPU 494.57 sec
2013-09-18 01:50:01,385 Stage-1 map = 76%, reduce = 0%, Cumulative CPU 494.57 sec
2013-09-18 01:50:02,403 Stage-1 map = 77%, reduce = 0%, Cumulative CPU 498.86 sec
2013-09-18 01:50:03,411 Stage-1 map = 80%, reduce = 0%, Cumulative CPU 506.12 sec
2013-09-18 01:50:04,419 Stage-1 map = 84%, reduce = 0%, Cumulative CPU 517.77 sec
2013-09-18 01:50:05,426 Stage-1 map = 85%, reduce = 0%, Cumulative CPU 529.34 sec
2013-09-18 01:50:06,432 Stage-1 map = 86%, reduce = 0%, Cumulative CPU 536.01 sec
2013-09-18 01:50:07,440 Stage-1 map = 89%, reduce = 0%, Cumulative CPU 574.29 sec
2013-09-18 01:50:08,448 Stage-1 map = 93%, reduce = 0%, Cumulative CPU 643.5 sec
2013-09-18 01:50:09,455 Stage-1 map = 95%, reduce = 0%, Cumulative CPU 690.76 sec
2013-09-18 01:50:10,461 Stage-1 map = 97%, reduce = 0%, Cumulative CPU 701.69 sec
2013-09-18 01:50:11,468 Stage-1 map = 98%, reduce = 0%, Cumulative CPU 701.69 sec
2013-09-18 01:50:12,491 Stage-1 map = 98%, reduce = 0%, Cumulative CPU 716.61 sec
2013-09-18 01:50:13,497 Stage-1 map = 99%, reduce = 4%, Cumulative CPU 758.65 sec
2013-09-18 01:50:14,502 Stage-1 map = 99%, reduce = 19%, Cumulative CPU 758.65 sec
2013-09-18 01:50:15,508 Stage-1 map = 100%, reduce = 21%, Cumulative CPU 774.74 sec
2013-09-18 01:50:16,515 Stage-1 map = 100%, reduce = 26%, Cumulative CPU 774.74 sec
2013-09-18 01:50:17,521 Stage-1 map = 100%, reduce = 28%, Cumulative CPU 774.74 sec
2013-09-18 01:50:18,526 Stage-1 map = 100%, reduce = 31%, Cumulative CPU 774.74 sec
2013-09-18 01:50:19,533 Stage-1 map = 100%, reduce = 32%, Cumulative CPU 774.74 sec
2013-09-18 01:50:20,539 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 774.74 sec
2013-09-18 01:50:21,545 Stage-1 map = 100%, reduce = 39%, Cumulative CPU 774.74 sec
2013-09-18 01:50:22,551 Stage-1 map = 100%, reduce = 42%, Cumulative CPU 774.74 sec
2013-09-18 01:50:23,557 Stage-1 map = 100%, reduce = 67%, Cumulative CPU 774.74 sec
2013-09-18 01:50:24,562 Stage-1 map = 100%, reduce = 69%, Cumulative CPU 774.74 sec
2013-09-18 01:50:25,568 Stage-1 map = 100%, reduce = 71%, Cumulative CPU 774.74 sec
2013-09-18 01:50:26,574 Stage-1 map = 100%, reduce = 82%, Cumulative CPU 774.74 sec
2013-09-18 01:50:27,583 Stage-1 map = 100%, reduce = 86%, Cumulative CPU 802.67 sec
2013-09-18 01:50:28,589 Stage-1 map = 100%, reduce = 94%, Cumulative CPU 874.81 sec
2013-09-18 01:50:29,594 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 934.02 sec
2013-09-18 01:50:30,617 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 934.02 sec
MapReduce Total cumulative CPU time: 15 minutes 34 seconds 20 msec
Ended Job = job_201309172235_0163
Launching Job 2 out of 2
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0164
Hadoop job information for Stage-2: number of mappers: 2; number of reducers: 1
2013-09-18 01:50:34,091 Stage-2 map = 0%, reduce = 0%
2013-09-18 01:50:44,120 Stage-2 map = 23%, reduce = 0%
2013-09-18 01:50:47,129 Stage-2 map = 46%, reduce = 0%
2013-09-18 01:50:51,142 Stage-2 map = 62%, reduce = 0%, Cumulative CPU 23.2 sec
2013-09-18 01:50:52,147 Stage-2 map = 62%, reduce = 0%, Cumulative CPU 23.2 sec
2013-09-18 01:50:53,152 Stage-2 map = 69%, reduce = 0%, Cumulative CPU 23.2 sec
2013-09-18 01:50:54,157 Stage-2 map = 69%, reduce = 0%, Cumulative CPU 23.2 sec
2013-09-18 01:50:55,162 Stage-2 map = 69%, reduce = 0%, Cumulative CPU 23.2 sec
2013-09-18 01:50:56,167 Stage-2 map = 75%, reduce = 0%, Cumulative CPU 23.2 sec
2013-09-18 01:50:58,116 Stage-2 map = 75%, reduce = 0%, Cumulative CPU 23.2 sec
2013-09-18 01:50:59,121 Stage-2 map = 81%, reduce = 17%, Cumulative CPU 23.2 sec
2013-09-18 01:51:00,129 Stage-2 map = 81%, reduce = 17%, Cumulative CPU 23.2 sec
2013-09-18 01:51:01,134 Stage-2 map = 81%, reduce = 17%, Cumulative CPU 57.42 sec
2013-09-18 01:51:02,139 Stage-2 map = 81%, reduce = 17%, Cumulative CPU 57.42 sec
2013-09-18 01:51:03,144 Stage-2 map = 81%, reduce = 17%, Cumulative CPU 57.42 sec
2013-09-18 01:51:04,149 Stage-2 map = 81%, reduce = 17%, Cumulative CPU 57.42 sec
2013-09-18 01:51:05,154 Stage-2 map = 87%, reduce = 17%, Cumulative CPU 57.42 sec
2013-09-18 01:51:06,158 Stage-2 map = 87%, reduce = 17%, Cumulative CPU 57.42 sec
2013-09-18 01:51:07,163 Stage-2 map = 87%, reduce = 17%, Cumulative CPU 57.42 sec
2013-09-18 01:51:08,172 Stage-2 map = 94%, reduce = 17%, Cumulative CPU 57.42 sec
2013-09-18 01:51:09,177 Stage-2 map = 94%, reduce = 17%, Cumulative CPU 57.42 sec
2013-09-18 01:51:10,182 Stage-2 map = 94%, reduce = 17%, Cumulative CPU 57.42 sec
2013-09-18 01:51:11,186 Stage-2 map = 94%, reduce = 17%, Cumulative CPU 57.42 sec
2013-09-18 01:51:12,191 Stage-2 map = 94%, reduce = 17%, Cumulative CPU 57.42 sec
2013-09-18 01:51:13,195 Stage-2 map = 94%, reduce = 17%, Cumulative CPU 57.42 sec
2013-09-18 01:51:14,200 Stage-2 map = 94%, reduce = 17%, Cumulative CPU 57.42 sec
2013-09-18 01:51:15,205 Stage-2 map = 94%, reduce = 17%, Cumulative CPU 57.42 sec
2013-09-18 01:51:16,209 Stage-2 map = 94%, reduce = 17%, Cumulative CPU 57.42 sec
2013-09-18 01:51:17,214 Stage-2 map = 100%, reduce = 17%, Cumulative CPU 77.78 sec
2013-09-18 01:51:18,218 Stage-2 map = 100%, reduce = 17%, Cumulative CPU 77.78 sec
2013-09-18 01:51:19,223 Stage-2 map = 100%, reduce = 17%, Cumulative CPU 77.78 sec
2013-09-18 01:51:20,227 Stage-2 map = 100%, reduce = 17%, Cumulative CPU 77.78 sec
2013-09-18 01:51:21,232 Stage-2 map = 100%, reduce = 17%, Cumulative CPU 77.78 sec
2013-09-18 01:51:22,236 Stage-2 map = 100%, reduce = 17%, Cumulative CPU 77.78 sec
2013-09-18 01:51:23,241 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 77.78 sec
2013-09-18 01:51:24,245 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 77.78 sec
2013-09-18 01:51:25,250 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 77.78 sec
2013-09-18 01:51:26,254 Stage-2 map = 100%, reduce = 89%, Cumulative CPU 77.78 sec
2013-09-18 01:51:27,259 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 87.04 sec
2013-09-18 01:51:28,264 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 87.04 sec
MapReduce Total cumulative CPU time: 1 minutes 27 seconds 40 msec
Ended Job = job_201309172235_0164
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 11 Cumulative CPU: 934.02 sec HDFS Read: 339202346 HDFS Write: 374792539 SUCCESS
Job 1: Map: 2 Reduce: 1 Cumulative CPU: 87.04 sec HDFS Read: 374796261 HDFS Write: 498 SUCCESS
Total MapReduce CPU Time Spent: 17 minutes 1 seconds 60 msec
OK
Time taken: 120.684 seconds, Fetched: 10 row(s)
hive> quit;
times: 2
query: SELECT ClientIP, ClientIP - 1, ClientIP - 2, ClientIP - 3, count(*) AS c FROM hits_100m GROUP BY ClientIP, ClientIP - 1, ClientIP - 2, ClientIP - 3 ORDER BY c DESC LIMIT 10;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_7222@mturlrep13_201309180151_945335118.txt
hive> SELECT ClientIP, ClientIP - 1, ClientIP - 2, ClientIP - 3, count(*) AS c FROM hits_100m GROUP BY ClientIP, ClientIP - 1, ClientIP - 2, ClientIP - 3 ORDER BY c DESC LIMIT 10;;
Total MapReduce jobs = 2
Launching Job 1 out of 2
Number of reduce tasks not specified. Estimated from input data size: 11
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0165
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 11
2013-09-18 01:51:37,818 Stage-1 map = 0%, reduce = 0%
2013-09-18 01:51:49,905 Stage-1 map = 2%, reduce = 0%
2013-09-18 01:51:50,923 Stage-1 map = 6%, reduce = 0%
2013-09-18 01:51:52,936 Stage-1 map = 15%, reduce = 0%
2013-09-18 01:51:53,959 Stage-1 map = 33%, reduce = 0%
2013-09-18 01:51:54,965 Stage-1 map = 36%, reduce = 0%
2013-09-18 01:51:55,979 Stage-1 map = 41%, reduce = 0%
2013-09-18 01:51:57,007 Stage-1 map = 62%, reduce = 0%
2013-09-18 01:51:58,018 Stage-1 map = 63%, reduce = 0%
2013-09-18 01:51:59,032 Stage-1 map = 66%, reduce = 0%, Cumulative CPU 13.91 sec
2013-09-18 01:52:00,049 Stage-1 map = 76%, reduce = 0%, Cumulative CPU 13.91 sec
2013-09-18 01:52:01,062 Stage-1 map = 79%, reduce = 0%, Cumulative CPU 546.5 sec
2013-09-18 01:52:02,075 Stage-1 map = 80%, reduce = 0%, Cumulative CPU 546.5 sec
2013-09-18 01:52:03,086 Stage-1 map = 81%, reduce = 0%, Cumulative CPU 553.11 sec
2013-09-18 01:52:04,094 Stage-1 map = 84%, reduce = 0%, Cumulative CPU 570.16 sec
2013-09-18 01:52:05,112 Stage-1 map = 86%, reduce = 0%, Cumulative CPU 591.73 sec
2013-09-18 01:52:06,122 Stage-1 map = 88%, reduce = 0%, Cumulative CPU 606.61 sec
2013-09-18 01:52:07,134 Stage-1 map = 94%, reduce = 0%, Cumulative CPU 689.49 sec
2013-09-18 01:52:08,140 Stage-1 map = 95%, reduce = 0%, Cumulative CPU 704.24 sec
2013-09-18 01:52:09,146 Stage-1 map = 95%, reduce = 0%, Cumulative CPU 704.24 sec
2013-09-18 01:52:10,152 Stage-1 map = 97%, reduce = 0%, Cumulative CPU 704.24 sec
2013-09-18 01:52:11,158 Stage-1 map = 97%, reduce = 0%, Cumulative CPU 704.24 sec
2013-09-18 01:52:12,165 Stage-1 map = 98%, reduce = 4%, Cumulative CPU 717.42 sec
2013-09-18 01:52:13,170 Stage-1 map = 99%, reduce = 23%, Cumulative CPU 745.46 sec
2013-09-18 01:52:14,176 Stage-1 map = 100%, reduce = 25%, Cumulative CPU 775.29 sec
2013-09-18 01:52:15,181 Stage-1 map = 100%, reduce = 29%, Cumulative CPU 775.29 sec
2013-09-18 01:52:16,187 Stage-1 map = 100%, reduce = 29%, Cumulative CPU 775.29 sec
2013-09-18 01:52:17,193 Stage-1 map = 100%, reduce = 29%, Cumulative CPU 775.29 sec
2013-09-18 01:52:18,199 Stage-1 map = 100%, reduce = 31%, Cumulative CPU 775.29 sec
2013-09-18 01:52:19,204 Stage-1 map = 100%, reduce = 31%, Cumulative CPU 775.29 sec
2013-09-18 01:52:20,209 Stage-1 map = 100%, reduce = 34%, Cumulative CPU 775.29 sec
2013-09-18 01:52:21,215 Stage-1 map = 100%, reduce = 43%, Cumulative CPU 775.29 sec
2013-09-18 01:52:22,221 Stage-1 map = 100%, reduce = 60%, Cumulative CPU 775.29 sec
2013-09-18 01:52:23,226 Stage-1 map = 100%, reduce = 67%, Cumulative CPU 775.29 sec
2013-09-18 01:52:24,232 Stage-1 map = 100%, reduce = 72%, Cumulative CPU 775.29 sec
2013-09-18 01:52:25,238 Stage-1 map = 100%, reduce = 79%, Cumulative CPU 775.29 sec
2013-09-18 01:52:26,245 Stage-1 map = 100%, reduce = 84%, Cumulative CPU 789.78 sec
2013-09-18 01:52:27,251 Stage-1 map = 100%, reduce = 91%, Cumulative CPU 848.54 sec
2013-09-18 01:52:28,257 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 936.77 sec
2013-09-18 01:52:29,275 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 936.77 sec
2013-09-18 01:52:30,282 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 936.77 sec
MapReduce Total cumulative CPU time: 15 minutes 36 seconds 770 msec
Ended Job = job_201309172235_0165
Launching Job 2 out of 2
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0166
Hadoop job information for Stage-2: number of mappers: 2; number of reducers: 1
2013-09-18 01:52:33,818 Stage-2 map = 0%, reduce = 0%
2013-09-18 01:52:43,850 Stage-2 map = 23%, reduce = 0%
2013-09-18 01:52:46,861 Stage-2 map = 46%, reduce = 0%
2013-09-18 01:52:49,871 Stage-2 map = 52%, reduce = 0%
2013-09-18 01:52:51,880 Stage-2 map = 69%, reduce = 0%, Cumulative CPU 24.11 sec
2013-09-18 01:52:52,885 Stage-2 map = 69%, reduce = 0%, Cumulative CPU 24.11 sec
2013-09-18 01:52:53,890 Stage-2 map = 69%, reduce = 0%, Cumulative CPU 24.11 sec
2013-09-18 01:52:54,896 Stage-2 map = 69%, reduce = 0%, Cumulative CPU 24.11 sec
2013-09-18 01:52:55,901 Stage-2 map = 75%, reduce = 0%, Cumulative CPU 24.11 sec
2013-09-18 01:52:56,907 Stage-2 map = 75%, reduce = 0%, Cumulative CPU 24.11 sec
2013-09-18 01:52:57,912 Stage-2 map = 75%, reduce = 0%, Cumulative CPU 24.11 sec
2013-09-18 01:52:58,918 Stage-2 map = 81%, reduce = 17%, Cumulative CPU 24.11 sec
2013-09-18 01:52:59,923 Stage-2 map = 81%, reduce = 17%, Cumulative CPU 24.11 sec
2013-09-18 01:53:00,928 Stage-2 map = 81%, reduce = 17%, Cumulative CPU 58.35 sec
2013-09-18 01:53:01,933 Stage-2 map = 87%, reduce = 17%, Cumulative CPU 58.35 sec
2013-09-18 01:53:02,938 Stage-2 map = 87%, reduce = 17%, Cumulative CPU 58.35 sec
2013-09-18 01:53:03,943 Stage-2 map = 87%, reduce = 17%, Cumulative CPU 58.35 sec
2013-09-18 01:53:04,948 Stage-2 map = 87%, reduce = 17%, Cumulative CPU 58.35 sec
2013-09-18 01:53:05,959 Stage-2 map = 87%, reduce = 17%, Cumulative CPU 58.35 sec
2013-09-18 01:53:06,963 Stage-2 map = 87%, reduce = 17%, Cumulative CPU 58.35 sec
2013-09-18 01:53:07,968 Stage-2 map = 94%, reduce = 17%, Cumulative CPU 58.35 sec
2013-09-18 01:53:08,973 Stage-2 map = 94%, reduce = 17%, Cumulative CPU 58.35 sec
2013-09-18 01:53:09,979 Stage-2 map = 94%, reduce = 17%, Cumulative CPU 58.35 sec
2013-09-18 01:53:10,984 Stage-2 map = 94%, reduce = 17%, Cumulative CPU 58.35 sec
2013-09-18 01:53:11,989 Stage-2 map = 94%, reduce = 17%, Cumulative CPU 58.35 sec
2013-09-18 01:53:12,995 Stage-2 map = 94%, reduce = 17%, Cumulative CPU 58.35 sec
2013-09-18 01:53:14,000 Stage-2 map = 94%, reduce = 17%, Cumulative CPU 58.35 sec
2013-09-18 01:53:15,006 Stage-2 map = 94%, reduce = 17%, Cumulative CPU 58.35 sec
2013-09-18 01:53:16,010 Stage-2 map = 100%, reduce = 17%, Cumulative CPU 78.21 sec
2013-09-18 01:53:17,014 Stage-2 map = 100%, reduce = 17%, Cumulative CPU 78.21 sec
2013-09-18 01:53:18,018 Stage-2 map = 100%, reduce = 17%, Cumulative CPU 78.21 sec
2013-09-18 01:53:19,023 Stage-2 map = 100%, reduce = 17%, Cumulative CPU 78.21 sec
2013-09-18 01:53:20,027 Stage-2 map = 100%, reduce = 17%, Cumulative CPU 78.21 sec
2013-09-18 01:53:21,031 Stage-2 map = 100%, reduce = 17%, Cumulative CPU 78.21 sec
2013-09-18 01:53:22,035 Stage-2 map = 100%, reduce = 17%, Cumulative CPU 78.21 sec
2013-09-18 01:53:23,039 Stage-2 map = 100%, reduce = 17%, Cumulative CPU 78.21 sec
2013-09-18 01:53:24,044 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 78.21 sec
2013-09-18 01:53:25,048 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 78.21 sec
2013-09-18 01:53:26,053 Stage-2 map = 100%, reduce = 83%, Cumulative CPU 78.21 sec
2013-09-18 01:53:27,058 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 87.43 sec
2013-09-18 01:53:28,063 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 87.43 sec
2013-09-18 01:53:29,068 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 87.43 sec
MapReduce Total cumulative CPU time: 1 minutes 27 seconds 430 msec
Ended Job = job_201309172235_0166
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 11 Cumulative CPU: 936.77 sec HDFS Read: 339202346 HDFS Write: 374792539 SUCCESS
Job 1: Map: 2 Reduce: 1 Cumulative CPU: 87.43 sec HDFS Read: 374796261 HDFS Write: 498 SUCCESS
Total MapReduce CPU Time Spent: 17 minutes 4 seconds 200 msec
OK
Time taken: 118.948 seconds, Fetched: 10 row(s)
hive> quit;
times: 3
query: SELECT ClientIP, ClientIP - 1, ClientIP - 2, ClientIP - 3, count(*) AS c FROM hits_100m GROUP BY ClientIP, ClientIP - 1, ClientIP - 2, ClientIP - 3 ORDER BY c DESC LIMIT 10;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_14110@mturlrep13_201309180153_1737492136.txt
hive> SELECT ClientIP, ClientIP - 1, ClientIP - 2, ClientIP - 3, count(*) AS c FROM hits_100m GROUP BY ClientIP, ClientIP - 1, ClientIP - 2, ClientIP - 3 ORDER BY c DESC LIMIT 10;;
Total MapReduce jobs = 2
Launching Job 1 out of 2
Number of reduce tasks not specified. Estimated from input data size: 11
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0167
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 11
2013-09-18 01:53:39,704 Stage-1 map = 0%, reduce = 0%
2013-09-18 01:53:50,787 Stage-1 map = 1%, reduce = 0%
2013-09-18 01:53:51,799 Stage-1 map = 4%, reduce = 0%
2013-09-18 01:53:52,813 Stage-1 map = 5%, reduce = 0%
2013-09-18 01:53:53,831 Stage-1 map = 14%, reduce = 0%
2013-09-18 01:53:54,840 Stage-1 map = 34%, reduce = 0%
2013-09-18 01:53:55,846 Stage-1 map = 35%, reduce = 0%
2013-09-18 01:53:56,851 Stage-1 map = 39%, reduce = 0%
2013-09-18 01:53:57,860 Stage-1 map = 59%, reduce = 0%
2013-09-18 01:53:58,883 Stage-1 map = 63%, reduce = 0%
2013-09-18 01:53:59,905 Stage-1 map = 64%, reduce = 0%, Cumulative CPU 13.8 sec
2013-09-18 01:54:00,943 Stage-1 map = 72%, reduce = 0%, Cumulative CPU 13.8 sec
2013-09-18 01:54:01,952 Stage-1 map = 80%, reduce = 0%, Cumulative CPU 535.34 sec
2013-09-18 01:54:02,959 Stage-1 map = 80%, reduce = 0%, Cumulative CPU 537.7 sec
2013-09-18 01:54:03,966 Stage-1 map = 82%, reduce = 0%, Cumulative CPU 540.16 sec
2013-09-18 01:54:04,973 Stage-1 map = 83%, reduce = 0%, Cumulative CPU 551.46 sec
2013-09-18 01:54:05,979 Stage-1 map = 85%, reduce = 0%, Cumulative CPU 573.16 sec
2013-09-18 01:54:07,008 Stage-1 map = 89%, reduce = 0%, Cumulative CPU 605.0 sec
2013-09-18 01:54:08,027 Stage-1 map = 92%, reduce = 0%, Cumulative CPU 654.51 sec
2013-09-18 01:54:09,034 Stage-1 map = 94%, reduce = 0%, Cumulative CPU 691.64 sec
2013-09-18 01:54:10,040 Stage-1 map = 96%, reduce = 0%, Cumulative CPU 700.45 sec
2013-09-18 01:54:11,047 Stage-1 map = 97%, reduce = 0%, Cumulative CPU 700.45 sec
2013-09-18 01:54:12,053 Stage-1 map = 97%, reduce = 0%, Cumulative CPU 700.45 sec
2013-09-18 01:54:13,060 Stage-1 map = 98%, reduce = 3%, Cumulative CPU 700.45 sec
2013-09-18 01:54:14,066 Stage-1 map = 99%, reduce = 21%, Cumulative CPU 745.67 sec
2013-09-18 01:54:15,072 Stage-1 map = 100%, reduce = 21%, Cumulative CPU 776.9 sec
2013-09-18 01:54:16,078 Stage-1 map = 100%, reduce = 29%, Cumulative CPU 776.9 sec
2013-09-18 01:54:17,084 Stage-1 map = 100%, reduce = 29%, Cumulative CPU 776.9 sec
2013-09-18 01:54:18,094 Stage-1 map = 100%, reduce = 29%, Cumulative CPU 776.9 sec
2013-09-18 01:54:19,103 Stage-1 map = 100%, reduce = 30%, Cumulative CPU 776.9 sec
2013-09-18 01:54:20,109 Stage-1 map = 100%, reduce = 30%, Cumulative CPU 776.9 sec
2013-09-18 01:54:21,115 Stage-1 map = 100%, reduce = 30%, Cumulative CPU 776.9 sec
2013-09-18 01:54:22,125 Stage-1 map = 100%, reduce = 43%, Cumulative CPU 776.9 sec
2013-09-18 01:54:23,131 Stage-1 map = 100%, reduce = 67%, Cumulative CPU 776.9 sec
2013-09-18 01:54:24,137 Stage-1 map = 100%, reduce = 67%, Cumulative CPU 776.9 sec
2013-09-18 01:54:25,144 Stage-1 map = 100%, reduce = 72%, Cumulative CPU 776.9 sec
2013-09-18 01:54:26,149 Stage-1 map = 100%, reduce = 82%, Cumulative CPU 776.9 sec
2013-09-18 01:54:27,155 Stage-1 map = 100%, reduce = 82%, Cumulative CPU 776.9 sec
2013-09-18 01:54:28,164 Stage-1 map = 100%, reduce = 92%, Cumulative CPU 862.86 sec
2013-09-18 01:54:29,177 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 934.72 sec
2013-09-18 01:54:30,183 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 934.72 sec
2013-09-18 01:54:31,188 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 934.72 sec
MapReduce Total cumulative CPU time: 15 minutes 34 seconds 720 msec
Ended Job = job_201309172235_0167
Launching Job 2 out of 2
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0168
Hadoop job information for Stage-2: number of mappers: 2; number of reducers: 1
2013-09-18 01:54:34,764 Stage-2 map = 0%, reduce = 0%
2013-09-18 01:54:44,797 Stage-2 map = 23%, reduce = 0%
2013-09-18 01:54:47,807 Stage-2 map = 46%, reduce = 0%
2013-09-18 01:54:50,817 Stage-2 map = 52%, reduce = 0%
2013-09-18 01:54:52,826 Stage-2 map = 69%, reduce = 0%, Cumulative CPU 23.57 sec
2013-09-18 01:54:53,831 Stage-2 map = 69%, reduce = 0%, Cumulative CPU 23.57 sec
2013-09-18 01:54:54,836 Stage-2 map = 69%, reduce = 0%, Cumulative CPU 23.57 sec
2013-09-18 01:54:55,841 Stage-2 map = 69%, reduce = 0%, Cumulative CPU 23.57 sec
2013-09-18 01:54:56,847 Stage-2 map = 75%, reduce = 0%, Cumulative CPU 23.57 sec
2013-09-18 01:54:57,852 Stage-2 map = 75%, reduce = 0%, Cumulative CPU 23.57 sec
2013-09-18 01:54:58,857 Stage-2 map = 75%, reduce = 0%, Cumulative CPU 23.57 sec
2013-09-18 01:54:59,863 Stage-2 map = 81%, reduce = 17%, Cumulative CPU 23.57 sec
2013-09-18 01:55:00,868 Stage-2 map = 81%, reduce = 17%, Cumulative CPU 23.57 sec
2013-09-18 01:55:01,874 Stage-2 map = 81%, reduce = 17%, Cumulative CPU 57.88 sec
2013-09-18 01:55:02,879 Stage-2 map = 87%, reduce = 17%, Cumulative CPU 57.88 sec
2013-09-18 01:55:03,883 Stage-2 map = 87%, reduce = 17%, Cumulative CPU 57.88 sec
2013-09-18 01:55:04,888 Stage-2 map = 87%, reduce = 17%, Cumulative CPU 57.88 sec
2013-09-18 01:55:05,898 Stage-2 map = 94%, reduce = 17%, Cumulative CPU 57.88 sec
2013-09-18 01:55:06,903 Stage-2 map = 94%, reduce = 17%, Cumulative CPU 57.88 sec
2013-09-18 01:55:07,908 Stage-2 map = 94%, reduce = 17%, Cumulative CPU 57.88 sec
2013-09-18 01:55:08,913 Stage-2 map = 94%, reduce = 17%, Cumulative CPU 57.88 sec
2013-09-18 01:55:09,918 Stage-2 map = 94%, reduce = 17%, Cumulative CPU 57.88 sec
2013-09-18 01:55:10,923 Stage-2 map = 94%, reduce = 17%, Cumulative CPU 57.88 sec
2013-09-18 01:55:11,928 Stage-2 map = 94%, reduce = 17%, Cumulative CPU 57.88 sec
2013-09-18 01:55:12,933 Stage-2 map = 94%, reduce = 17%, Cumulative CPU 57.88 sec
2013-09-18 01:55:13,938 Stage-2 map = 94%, reduce = 17%, Cumulative CPU 57.88 sec
2013-09-18 01:55:14,942 Stage-2 map = 94%, reduce = 17%, Cumulative CPU 57.88 sec
2013-09-18 01:55:15,947 Stage-2 map = 94%, reduce = 17%, Cumulative CPU 57.88 sec
2013-09-18 01:55:16,952 Stage-2 map = 100%, reduce = 17%, Cumulative CPU 77.06 sec
2013-09-18 01:55:17,957 Stage-2 map = 100%, reduce = 17%, Cumulative CPU 77.06 sec
2013-09-18 01:55:18,962 Stage-2 map = 100%, reduce = 17%, Cumulative CPU 77.06 sec
2013-09-18 01:55:19,967 Stage-2 map = 100%, reduce = 17%, Cumulative CPU 77.06 sec
2013-09-18 01:55:20,972 Stage-2 map = 100%, reduce = 17%, Cumulative CPU 77.06 sec
2013-09-18 01:55:21,977 Stage-2 map = 100%, reduce = 17%, Cumulative CPU 77.06 sec
2013-09-18 01:55:22,982 Stage-2 map = 100%, reduce = 17%, Cumulative CPU 77.06 sec
2013-09-18 01:55:23,987 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 77.06 sec
2013-09-18 01:55:24,992 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 77.06 sec
2013-09-18 01:55:25,997 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 77.06 sec
2013-09-18 01:55:27,002 Stage-2 map = 100%, reduce = 89%, Cumulative CPU 77.06 sec
2013-09-18 01:55:28,008 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 86.19 sec
2013-09-18 01:55:29,013 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 86.19 sec
MapReduce Total cumulative CPU time: 1 minutes 26 seconds 190 msec
Ended Job = job_201309172235_0168
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 11 Cumulative CPU: 934.72 sec HDFS Read: 339202346 HDFS Write: 374792539 SUCCESS
Job 1: Map: 2 Reduce: 1 Cumulative CPU: 86.19 sec HDFS Read: 374796261 HDFS Write: 498 SUCCESS
Total MapReduce CPU Time Spent: 17 minutes 0 seconds 910 msec
OK
Time taken: 118.18 seconds, Fetched: 10 row(s)
hive> quit;
times: 1
query:
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_21734@mturlrep13_201309180155_15887287.txt
hive> ;
hive> quit;
times: 2
query:
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_21941@mturlrep13_201309180155_1922409770.txt
hive> ;
hive> quit;
times: 3
query:
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_22141@mturlrep13_201309180155_818952348.txt
hive> ;
hive> quit;
times: 1
query: SELECT URL, count(*) AS PageViews FROM hits_100m WHERE CounterID = 34 AND EventDate >= TIMESTAMP('2013-07-01') AND EventDate <= TIMESTAMP('2013-07-31') AND NOT DontCountHits != 0 AND NOT Refresh != 0 AND URL != '' GROUP BY URL ORDER BY PageViews DESC LIMIT 10;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_22349@mturlrep13_201309180155_129320467.txt
hive> SELECT URL, count(*) AS PageViews FROM hits_100m WHERE CounterID = 34 AND EventDate >= TIMESTAMP('2013-07-01') AND EventDate <= TIMESTAMP('2013-07-31') AND NOT DontCountHits != 0 AND NOT Refresh != 0 AND URL != '' GROUP BY URL ORDER BY PageViews DESC LIMIT 10;;
Total MapReduce jobs = 2
Launching Job 1 out of 2
Number of reduce tasks not specified. Estimated from input data size: 11
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0169
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 11
2013-09-18 01:55:58,117 Stage-1 map = 0%, reduce = 0%
2013-09-18 01:56:11,233 Stage-1 map = 2%, reduce = 0%
2013-09-18 01:56:12,984 Stage-1 map = 3%, reduce = 0%
2013-09-18 01:56:13,990 Stage-1 map = 8%, reduce = 0%
2013-09-18 01:56:14,996 Stage-1 map = 14%, reduce = 0%
2013-09-18 01:56:16,314 Stage-1 map = 16%, reduce = 0%
2013-09-18 01:56:21,716 Stage-1 map = 36%, reduce = 0%
2013-09-18 01:56:22,731 Stage-1 map = 38%, reduce = 0%
2013-09-18 01:56:24,752 Stage-1 map = 52%, reduce = 0%, Cumulative CPU 22.96 sec
2013-09-18 01:56:25,760 Stage-1 map = 58%, reduce = 0%, Cumulative CPU 22.96 sec
2013-09-18 01:56:26,767 Stage-1 map = 58%, reduce = 0%, Cumulative CPU 22.96 sec
2013-09-18 01:56:28,182 Stage-1 map = 69%, reduce = 0%, Cumulative CPU 44.33 sec
2013-09-18 01:56:29,188 Stage-1 map = 69%, reduce = 0%, Cumulative CPU 44.33 sec
2013-09-18 01:56:30,404 Stage-1 map = 75%, reduce = 0%, Cumulative CPU 90.25 sec
2013-09-18 01:56:31,813 Stage-1 map = 93%, reduce = 0%, Cumulative CPU 335.9 sec
2013-09-18 01:56:32,821 Stage-1 map = 98%, reduce = 0%, Cumulative CPU 404.3 sec
2013-09-18 01:56:33,827 Stage-1 map = 100%, reduce = 2%, Cumulative CPU 427.67 sec
2013-09-18 01:56:34,833 Stage-1 map = 100%, reduce = 11%, Cumulative CPU 427.67 sec
2013-09-18 01:56:35,840 Stage-1 map = 100%, reduce = 14%, Cumulative CPU 427.67 sec
2013-09-18 01:56:36,846 Stage-1 map = 100%, reduce = 23%, Cumulative CPU 427.67 sec
2013-09-18 01:56:37,854 Stage-1 map = 100%, reduce = 41%, Cumulative CPU 436.66 sec
2013-09-18 01:56:38,862 Stage-1 map = 100%, reduce = 47%, Cumulative CPU 436.66 sec
2013-09-18 01:56:39,869 Stage-1 map = 100%, reduce = 69%, Cumulative CPU 446.83 sec
2013-09-18 01:56:40,875 Stage-1 map = 100%, reduce = 82%, Cumulative CPU 454.28 sec
2013-09-18 01:56:41,881 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 463.94 sec
2013-09-18 01:56:42,887 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 463.94 sec
MapReduce Total cumulative CPU time: 7 minutes 43 seconds 940 msec
Ended Job = job_201309172235_0169
Launching Job 2 out of 2
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0170
Hadoop job information for Stage-2: number of mappers: 1; number of reducers: 1
2013-09-18 01:56:46,407 Stage-2 map = 0%, reduce = 0%
2013-09-18 01:56:48,416 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.68 sec
2013-09-18 01:56:49,421 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.68 sec
2013-09-18 01:56:50,426 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.68 sec
2013-09-18 01:56:51,430 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.68 sec
2013-09-18 01:56:52,435 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.68 sec
2013-09-18 01:56:53,439 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.68 sec
2013-09-18 01:56:54,444 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.68 sec
2013-09-18 01:56:55,450 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 2.14 sec
2013-09-18 01:56:56,455 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 2.14 sec
2013-09-18 01:56:57,460 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 2.14 sec
MapReduce Total cumulative CPU time: 2 seconds 140 msec
Ended Job = job_201309172235_0170
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 11 Cumulative CPU: 463.94 sec HDFS Read: 993495605 HDFS Write: 1056 SUCCESS
Job 1: Map: 1 Reduce: 1 Cumulative CPU: 2.14 sec HDFS Read: 4633 HDFS Write: 0 SUCCESS
Total MapReduce CPU Time Spent: 7 minutes 46 seconds 80 msec
OK
Time taken: 70.292 seconds
hive> quit;
times: 2
query: SELECT URL, count(*) AS PageViews FROM hits_100m WHERE CounterID = 34 AND EventDate >= TIMESTAMP('2013-07-01') AND EventDate <= TIMESTAMP('2013-07-31') AND NOT DontCountHits != 0 AND NOT Refresh != 0 AND URL != '' GROUP BY URL ORDER BY PageViews DESC LIMIT 10;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_29743@mturlrep13_201309180156_208621189.txt
hive> SELECT URL, count(*) AS PageViews FROM hits_100m WHERE CounterID = 34 AND EventDate >= TIMESTAMP('2013-07-01') AND EventDate <= TIMESTAMP('2013-07-31') AND NOT DontCountHits != 0 AND NOT Refresh != 0 AND URL != '' GROUP BY URL ORDER BY PageViews DESC LIMIT 10;;
Total MapReduce jobs = 2
Launching Job 1 out of 2
Number of reduce tasks not specified. Estimated from input data size: 11
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0171
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 11
2013-09-18 01:57:08,252 Stage-1 map = 0%, reduce = 0%
2013-09-18 01:57:17,362 Stage-1 map = 1%, reduce = 0%
2013-09-18 01:57:18,402 Stage-1 map = 8%, reduce = 0%
2013-09-18 01:57:19,415 Stage-1 map = 9%, reduce = 0%
2013-09-18 01:57:20,427 Stage-1 map = 13%, reduce = 0%
2013-09-18 01:57:21,432 Stage-1 map = 36%, reduce = 0%
2013-09-18 01:57:22,447 Stage-1 map = 40%, reduce = 0%, Cumulative CPU 8.67 sec
2013-09-18 01:57:23,455 Stage-1 map = 46%, reduce = 0%, Cumulative CPU 8.67 sec
2013-09-18 01:57:24,469 Stage-1 map = 52%, reduce = 0%, Cumulative CPU 8.67 sec
2013-09-18 01:57:25,476 Stage-1 map = 64%, reduce = 0%, Cumulative CPU 33.09 sec
2013-09-18 01:57:26,484 Stage-1 map = 88%, reduce = 0%, Cumulative CPU 276.39 sec
2013-09-18 01:57:27,491 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 454.8 sec
2013-09-18 01:57:28,497 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 454.8 sec
2013-09-18 01:57:29,504 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 454.8 sec
2013-09-18 01:57:30,510 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 454.8 sec
2013-09-18 01:57:31,516 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 454.8 sec
2013-09-18 01:57:32,522 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 454.8 sec
2013-09-18 01:57:33,529 Stage-1 map = 100%, reduce = 6%, Cumulative CPU 454.8 sec
2013-09-18 01:57:34,536 Stage-1 map = 100%, reduce = 18%, Cumulative CPU 454.8 sec
2013-09-18 01:57:35,549 Stage-1 map = 100%, reduce = 52%, Cumulative CPU 469.58 sec
2013-09-18 01:57:36,556 Stage-1 map = 100%, reduce = 88%, Cumulative CPU 487.12 sec
2013-09-18 01:57:37,564 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 493.48 sec
2013-09-18 01:57:38,570 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 493.48 sec
2013-09-18 01:57:39,576 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 493.48 sec
MapReduce Total cumulative CPU time: 8 minutes 13 seconds 480 msec
Ended Job = job_201309172235_0171
Launching Job 2 out of 2
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0172
Hadoop job information for Stage-2: number of mappers: 1; number of reducers: 1
2013-09-18 01:57:43,495 Stage-2 map = 0%, reduce = 0%
2013-09-18 01:57:44,501 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.78 sec
2013-09-18 01:57:45,506 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.78 sec
2013-09-18 01:57:46,511 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.78 sec
2013-09-18 01:57:47,516 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.78 sec
2013-09-18 01:57:48,520 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.78 sec
2013-09-18 01:57:49,524 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.78 sec
2013-09-18 01:57:50,529 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.78 sec
2013-09-18 01:57:51,533 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.78 sec
2013-09-18 01:57:52,538 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 2.25 sec
2013-09-18 01:57:53,544 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 2.25 sec
2013-09-18 01:57:54,549 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 2.25 sec
MapReduce Total cumulative CPU time: 2 seconds 250 msec
Ended Job = job_201309172235_0172
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 11 Cumulative CPU: 493.48 sec HDFS Read: 993495605 HDFS Write: 1056 SUCCESS
Job 1: Map: 1 Reduce: 1 Cumulative CPU: 2.25 sec HDFS Read: 4633 HDFS Write: 0 SUCCESS
Total MapReduce CPU Time Spent: 8 minutes 15 seconds 730 msec
OK
Time taken: 55.211 seconds
hive> quit;
times: 3
query: SELECT URL, count(*) AS PageViews FROM hits_100m WHERE CounterID = 34 AND EventDate >= TIMESTAMP('2013-07-01') AND EventDate <= TIMESTAMP('2013-07-31') AND NOT DontCountHits != 0 AND NOT Refresh != 0 AND URL != '' GROUP BY URL ORDER BY PageViews DESC LIMIT 10;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_4778@mturlrep13_201309180157_2071772922.txt
hive> SELECT URL, count(*) AS PageViews FROM hits_100m WHERE CounterID = 34 AND EventDate >= TIMESTAMP('2013-07-01') AND EventDate <= TIMESTAMP('2013-07-31') AND NOT DontCountHits != 0 AND NOT Refresh != 0 AND URL != '' GROUP BY URL ORDER BY PageViews DESC LIMIT 10;;
Total MapReduce jobs = 2
Launching Job 1 out of 2
Number of reduce tasks not specified. Estimated from input data size: 11
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0173
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 11
2013-09-18 01:58:05,573 Stage-1 map = 0%, reduce = 0%
2013-09-18 01:58:14,649 Stage-1 map = 1%, reduce = 0%
2013-09-18 01:58:15,664 Stage-1 map = 9%, reduce = 0%
2013-09-18 01:58:17,723 Stage-1 map = 12%, reduce = 0%
2013-09-18 01:58:18,734 Stage-1 map = 36%, reduce = 0%
2013-09-18 01:58:19,756 Stage-1 map = 39%, reduce = 0%, Cumulative CPU 8.38 sec
2013-09-18 01:58:20,765 Stage-1 map = 43%, reduce = 0%, Cumulative CPU 8.38 sec
2013-09-18 01:58:21,773 Stage-1 map = 53%, reduce = 0%, Cumulative CPU 8.38 sec
2013-09-18 01:58:22,797 Stage-1 map = 63%, reduce = 0%, Cumulative CPU 32.26 sec
2013-09-18 01:58:23,804 Stage-1 map = 87%, reduce = 0%, Cumulative CPU 254.83 sec
2013-09-18 01:58:24,812 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 460.38 sec
2013-09-18 01:58:25,818 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 460.38 sec
2013-09-18 01:58:26,825 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 460.38 sec
2013-09-18 01:58:27,831 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 460.38 sec
2013-09-18 01:58:28,836 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 460.38 sec
2013-09-18 01:58:29,842 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 460.38 sec
2013-09-18 01:58:30,847 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 460.38 sec
2013-09-18 01:58:31,854 Stage-1 map = 100%, reduce = 15%, Cumulative CPU 460.38 sec
2013-09-18 01:58:32,864 Stage-1 map = 100%, reduce = 52%, Cumulative CPU 474.21 sec
2013-09-18 01:58:33,870 Stage-1 map = 100%, reduce = 82%, Cumulative CPU 487.03 sec
2013-09-18 01:58:34,876 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 496.96 sec
2013-09-18 01:58:35,882 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 496.96 sec
2013-09-18 01:58:36,888 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 496.96 sec
MapReduce Total cumulative CPU time: 8 minutes 16 seconds 960 msec
Ended Job = job_201309172235_0173
Launching Job 2 out of 2
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0174
Hadoop job information for Stage-2: number of mappers: 1; number of reducers: 1
2013-09-18 01:58:40,381 Stage-2 map = 0%, reduce = 0%
2013-09-18 01:58:41,387 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.7 sec
2013-09-18 01:58:42,393 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.7 sec
2013-09-18 01:58:43,398 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.7 sec
2013-09-18 01:58:44,403 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.7 sec
2013-09-18 01:58:45,408 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.7 sec
2013-09-18 01:58:46,413 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.7 sec
2013-09-18 01:58:47,418 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.7 sec
2013-09-18 01:58:48,423 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.7 sec
2013-09-18 01:58:49,429 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 2.2 sec
2013-09-18 01:58:50,435 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 2.2 sec
2013-09-18 01:58:51,440 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 2.2 sec
MapReduce Total cumulative CPU time: 2 seconds 200 msec
Ended Job = job_201309172235_0174
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 11 Cumulative CPU: 496.96 sec HDFS Read: 993495605 HDFS Write: 1056 SUCCESS
Job 1: Map: 1 Reduce: 1 Cumulative CPU: 2.2 sec HDFS Read: 4633 HDFS Write: 0 SUCCESS
Total MapReduce CPU Time Spent: 8 minutes 19 seconds 160 msec
OK
Time taken: 55.076 seconds
hive> quit;
times: 1
query: SELECT Title, count(*) AS PageViews FROM hits_100m WHERE CounterID = 34 AND EventDate >= TIMESTAMP('2013-07-01') AND EventDate <= TIMESTAMP('2013-07-31') AND NOT DontCountHits != 0 AND NOT Refresh != 0 AND Title != '' GROUP BY Title ORDER BY PageViews DESC LIMIT 10;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_12570@mturlrep13_201309180158_2097047961.txt
hive> SELECT Title, count(*) AS PageViews FROM hits_100m WHERE CounterID = 34 AND EventDate >= TIMESTAMP('2013-07-01') AND EventDate <= TIMESTAMP('2013-07-31') AND NOT DontCountHits != 0 AND NOT Refresh != 0 AND Title != '' GROUP BY Title ORDER BY PageViews DESC LIMIT 10;;
Total MapReduce jobs = 2
Launching Job 1 out of 2
Number of reduce tasks not specified. Estimated from input data size: 11
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0175
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 11
2013-09-18 01:59:09,511 Stage-1 map = 0%, reduce = 0%
2013-09-18 01:59:19,576 Stage-1 map = 5%, reduce = 0%
2013-09-18 01:59:22,593 Stage-1 map = 15%, reduce = 0%
2013-09-18 01:59:25,607 Stage-1 map = 29%, reduce = 0%
2013-09-18 01:59:26,612 Stage-1 map = 30%, reduce = 0%
2013-09-18 01:59:28,833 Stage-1 map = 42%, reduce = 0%
2013-09-18 01:59:34,330 Stage-1 map = 67%, reduce = 0%, Cumulative CPU 122.09 sec
2013-09-18 01:59:35,337 Stage-1 map = 77%, reduce = 0%, Cumulative CPU 176.65 sec
2013-09-18 01:59:36,349 Stage-1 map = 82%, reduce = 0%, Cumulative CPU 230.21 sec
2013-09-18 01:59:37,356 Stage-1 map = 99%, reduce = 0%, Cumulative CPU 395.83 sec
2013-09-18 01:59:38,362 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 407.07 sec
2013-09-18 01:59:39,368 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 407.07 sec
2013-09-18 01:59:40,374 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 407.07 sec
2013-09-18 01:59:41,380 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 407.07 sec
2013-09-18 01:59:42,386 Stage-1 map = 100%, reduce = 6%, Cumulative CPU 407.07 sec
2013-09-18 01:59:43,395 Stage-1 map = 100%, reduce = 30%, Cumulative CPU 413.7 sec
2013-09-18 01:59:44,403 Stage-1 map = 100%, reduce = 61%, Cumulative CPU 423.63 sec
2013-09-18 01:59:45,410 Stage-1 map = 100%, reduce = 82%, Cumulative CPU 434.03 sec
2013-09-18 01:59:46,417 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 443.68 sec
2013-09-18 01:59:47,424 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 443.68 sec
MapReduce Total cumulative CPU time: 7 minutes 23 seconds 680 msec
Ended Job = job_201309172235_0175
Launching Job 2 out of 2
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0176
Hadoop job information for Stage-2: number of mappers: 1; number of reducers: 1
2013-09-18 01:59:50,937 Stage-2 map = 0%, reduce = 0%
2013-09-18 01:59:52,947 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.72 sec
2013-09-18 01:59:53,953 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.72 sec
2013-09-18 01:59:54,958 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.72 sec
2013-09-18 01:59:55,963 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.72 sec
2013-09-18 01:59:56,969 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.72 sec
2013-09-18 01:59:57,974 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.72 sec
2013-09-18 01:59:58,981 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.72 sec
2013-09-18 01:59:59,987 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 0.72 sec
2013-09-18 02:00:00,993 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 2.25 sec
2013-09-18 02:00:01,999 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 2.25 sec
MapReduce Total cumulative CPU time: 2 seconds 250 msec
Ended Job = job_201309172235_0176
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 11 Cumulative CPU: 443.68 sec HDFS Read: 876843443 HDFS Write: 1056 SUCCESS
Job 1: Map: 1 Reduce: 1 Cumulative CPU: 2.25 sec HDFS Read: 4633 HDFS Write: 0 SUCCESS
Total MapReduce CPU Time Spent: 7 minutes 25 seconds 930 msec
OK
Time taken: 63.4 seconds
hive> quit;
times: 2
query: SELECT Title, count(*) AS PageViews FROM hits_100m WHERE CounterID = 34 AND EventDate >= TIMESTAMP('2013-07-01') AND EventDate <= TIMESTAMP('2013-07-31') AND NOT DontCountHits != 0 AND NOT Refresh != 0 AND Title != '' GROUP BY Title ORDER BY PageViews DESC LIMIT 10;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_19761@mturlrep13_201309180200_1379051620.txt
hive> SELECT Title, count(*) AS PageViews FROM hits_100m WHERE CounterID = 34 AND EventDate >= TIMESTAMP('2013-07-01') AND EventDate <= TIMESTAMP('2013-07-31') AND NOT DontCountHits != 0 AND NOT Refresh != 0 AND Title != '' GROUP BY Title ORDER BY PageViews DESC LIMIT 10;;
Total MapReduce jobs = 2
Launching Job 1 out of 2
Number of reduce tasks not specified. Estimated from input data size: 11
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0177
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 11
2013-09-18 02:00:13,526 Stage-1 map = 0%, reduce = 0%
2013-09-18 02:00:23,659 Stage-1 map = 11%, reduce = 0%
2013-09-18 02:00:26,730 Stage-1 map = 48%, reduce = 0%
2013-09-18 02:00:28,741 Stage-1 map = 50%, reduce = 0%
2013-09-18 02:00:29,755 Stage-1 map = 66%, reduce = 0%, Cumulative CPU 48.44 sec
2013-09-18 02:00:30,772 Stage-1 map = 77%, reduce = 0%, Cumulative CPU 100.32 sec
2013-09-18 02:00:31,784 Stage-1 map = 91%, reduce = 0%, Cumulative CPU 281.43 sec
2013-09-18 02:00:32,792 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 435.17 sec
2013-09-18 02:00:33,799 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 435.17 sec
2013-09-18 02:00:34,806 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 435.17 sec
2013-09-18 02:00:35,812 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 435.17 sec
2013-09-18 02:00:36,818 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 435.17 sec
2013-09-18 02:00:37,824 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 435.17 sec
2013-09-18 02:00:38,831 Stage-1 map = 100%, reduce = 15%, Cumulative CPU 435.17 sec
2013-09-18 02:00:39,843 Stage-1 map = 100%, reduce = 58%, Cumulative CPU 448.52 sec
2013-09-18 02:00:40,850 Stage-1 map = 100%, reduce = 94%, Cumulative CPU 470.07 sec
2013-09-18 02:00:41,857 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 473.43 sec
2013-09-18 02:00:42,864 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 473.43 sec
MapReduce Total cumulative CPU time: 7 minutes 53 seconds 430 msec
Ended Job = job_201309172235_0177
Launching Job 2 out of 2
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0178
Hadoop job information for Stage-2: number of mappers: 1; number of reducers: 1
2013-09-18 02:00:45,348 Stage-2 map = 0%, reduce = 0%
2013-09-18 02:00:47,357 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.71 sec
2013-09-18 02:00:48,363 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.71 sec
2013-09-18 02:00:49,367 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.71 sec
2013-09-18 02:00:50,372 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.71 sec
2013-09-18 02:00:51,376 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.71 sec
2013-09-18 02:00:52,381 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.71 sec
2013-09-18 02:00:53,385 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.71 sec
2013-09-18 02:00:54,390 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 0.71 sec
2013-09-18 02:00:55,396 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 2.19 sec
2013-09-18 02:00:56,403 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 2.19 sec
MapReduce Total cumulative CPU time: 2 seconds 190 msec
Ended Job = job_201309172235_0178
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 11 Cumulative CPU: 473.43 sec HDFS Read: 876843443 HDFS Write: 1056 SUCCESS
Job 1: Map: 1 Reduce: 1 Cumulative CPU: 2.19 sec HDFS Read: 4633 HDFS Write: 0 SUCCESS
Total MapReduce CPU Time Spent: 7 minutes 55 seconds 620 msec
OK
Time taken: 52.093 seconds
hive> quit;
times: 3
query: SELECT Title, count(*) AS PageViews FROM hits_100m WHERE CounterID = 34 AND EventDate >= TIMESTAMP('2013-07-01') AND EventDate <= TIMESTAMP('2013-07-31') AND NOT DontCountHits != 0 AND NOT Refresh != 0 AND Title != '' GROUP BY Title ORDER BY PageViews DESC LIMIT 10;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_27543@mturlrep13_201309180200_2073175542.txt
hive> SELECT Title, count(*) AS PageViews FROM hits_100m WHERE CounterID = 34 AND EventDate >= TIMESTAMP('2013-07-01') AND EventDate <= TIMESTAMP('2013-07-31') AND NOT DontCountHits != 0 AND NOT Refresh != 0 AND Title != '' GROUP BY Title ORDER BY PageViews DESC LIMIT 10;;
Total MapReduce jobs = 2
Launching Job 1 out of 2
Number of reduce tasks not specified. Estimated from input data size: 11
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0179
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 11
2013-09-18 02:01:07,075 Stage-1 map = 0%, reduce = 0%
2013-09-18 02:01:16,231 Stage-1 map = 4%, reduce = 0%
2013-09-18 02:01:17,250 Stage-1 map = 10%, reduce = 0%
2013-09-18 02:01:19,265 Stage-1 map = 22%, reduce = 0%
2013-09-18 02:01:20,281 Stage-1 map = 48%, reduce = 0%
2013-09-18 02:01:21,287 Stage-1 map = 50%, reduce = 0%
2013-09-18 02:01:22,302 Stage-1 map = 54%, reduce = 0%, Cumulative CPU 28.59 sec
2013-09-18 02:01:23,344 Stage-1 map = 69%, reduce = 0%, Cumulative CPU 58.45 sec
2013-09-18 02:01:24,364 Stage-1 map = 76%, reduce = 0%, Cumulative CPU 92.94 sec
2013-09-18 02:01:25,375 Stage-1 map = 93%, reduce = 0%, Cumulative CPU 322.31 sec
2013-09-18 02:01:26,382 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 437.99 sec
2013-09-18 02:01:27,387 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 437.99 sec
2013-09-18 02:01:28,392 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 437.99 sec
2013-09-18 02:01:29,398 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 437.99 sec
2013-09-18 02:01:30,404 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 437.99 sec
2013-09-18 02:01:31,410 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 437.99 sec
2013-09-18 02:01:32,417 Stage-1 map = 100%, reduce = 24%, Cumulative CPU 437.99 sec
2013-09-18 02:01:33,427 Stage-1 map = 100%, reduce = 52%, Cumulative CPU 448.33 sec
2013-09-18 02:01:34,433 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 477.42 sec
2013-09-18 02:01:35,440 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 477.42 sec
2013-09-18 02:01:36,447 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 477.42 sec
MapReduce Total cumulative CPU time: 7 minutes 57 seconds 420 msec
Ended Job = job_201309172235_0179
Launching Job 2 out of 2
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0180
Hadoop job information for Stage-2: number of mappers: 1; number of reducers: 1
2013-09-18 02:01:39,908 Stage-2 map = 0%, reduce = 0%
2013-09-18 02:01:40,913 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.79 sec
2013-09-18 02:01:41,919 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.79 sec
2013-09-18 02:01:42,924 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.79 sec
2013-09-18 02:01:43,928 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.79 sec
2013-09-18 02:01:44,933 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.79 sec
2013-09-18 02:01:46,032 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.79 sec
2013-09-18 02:01:47,037 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.79 sec
2013-09-18 02:01:48,042 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 0.79 sec
2013-09-18 02:01:49,048 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 2.21 sec
2013-09-18 02:01:50,053 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 2.21 sec
2013-09-18 02:01:51,058 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 2.21 sec
MapReduce Total cumulative CPU time: 2 seconds 210 msec
Ended Job = job_201309172235_0180
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 11 Cumulative CPU: 477.42 sec HDFS Read: 876843443 HDFS Write: 1056 SUCCESS
Job 1: Map: 1 Reduce: 1 Cumulative CPU: 2.21 sec HDFS Read: 4633 HDFS Write: 0 SUCCESS
Total MapReduce CPU Time Spent: 7 minutes 59 seconds 630 msec
OK
Time taken: 52.895 seconds
hive> quit;
times: 1
query: SELECT URL, count(*) AS PageViews FROM hits_100m WHERE CounterID = 34 AND EventDate >= TIMESTAMP('2013-07-01') AND EventDate <= TIMESTAMP('2013-07-31') AND NOT Refresh != 0 AND IsLink != 0 AND NOT IsDownload != 0 GROUP BY URL ORDER BY PageViews DESC LIMIT 1000;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_2326@mturlrep13_201309180201_303775162.txt
hive> SELECT URL, count(*) AS PageViews FROM hits_100m WHERE CounterID = 34 AND EventDate >= TIMESTAMP('2013-07-01') AND EventDate <= TIMESTAMP('2013-07-31') AND NOT Refresh != 0 AND IsLink != 0 AND NOT IsDownload != 0 GROUP BY URL ORDER BY PageViews DESC LIMIT 1000;;
Total MapReduce jobs = 2
Launching Job 1 out of 2
Number of reduce tasks not specified. Estimated from input data size: 11
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0181
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 11
2013-09-18 02:02:08,645 Stage-1 map = 0%, reduce = 0%
2013-09-18 02:02:18,769 Stage-1 map = 1%, reduce = 0%
2013-09-18 02:02:22,796 Stage-1 map = 5%, reduce = 0%
2013-09-18 02:02:23,803 Stage-1 map = 6%, reduce = 0%
2013-09-18 02:02:27,959 Stage-1 map = 24%, reduce = 0%
2013-09-18 02:02:32,561 Stage-1 map = 44%, reduce = 0%
2013-09-18 02:02:33,569 Stage-1 map = 54%, reduce = 0%
2013-09-18 02:02:34,585 Stage-1 map = 61%, reduce = 0%, Cumulative CPU 56.79 sec
2013-09-18 02:02:35,594 Stage-1 map = 62%, reduce = 0%, Cumulative CPU 80.27 sec
2013-09-18 02:02:38,969 Stage-1 map = 74%, reduce = 0%, Cumulative CPU 148.88 sec
2013-09-18 02:02:39,974 Stage-1 map = 79%, reduce = 0%, Cumulative CPU 219.04 sec
2013-09-18 02:02:40,981 Stage-1 map = 92%, reduce = 0%, Cumulative CPU 315.44 sec
2013-09-18 02:02:41,987 Stage-1 map = 99%, reduce = 0%, Cumulative CPU 427.42 sec
2013-09-18 02:02:42,993 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 438.91 sec
2013-09-18 02:02:44,000 Stage-1 map = 100%, reduce = 3%, Cumulative CPU 438.91 sec
2013-09-18 02:02:45,006 Stage-1 map = 100%, reduce = 12%, Cumulative CPU 438.91 sec
2013-09-18 02:02:46,014 Stage-1 map = 100%, reduce = 45%, Cumulative CPU 454.78 sec
2013-09-18 02:02:47,020 Stage-1 map = 100%, reduce = 55%, Cumulative CPU 454.78 sec
2013-09-18 02:02:48,027 Stage-1 map = 100%, reduce = 82%, Cumulative CPU 465.13 sec
2013-09-18 02:02:49,033 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 474.77 sec
2013-09-18 02:02:50,040 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 474.77 sec
2013-09-18 02:02:51,046 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 474.77 sec
MapReduce Total cumulative CPU time: 7 minutes 54 seconds 770 msec
Ended Job = job_201309172235_0181
Launching Job 2 out of 2
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0182
Hadoop job information for Stage-2: number of mappers: 1; number of reducers: 1
2013-09-18 02:02:54,504 Stage-2 map = 0%, reduce = 0%
2013-09-18 02:02:56,511 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.71 sec
2013-09-18 02:02:57,516 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.71 sec
2013-09-18 02:02:58,520 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.71 sec
2013-09-18 02:02:59,524 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.71 sec
2013-09-18 02:03:00,529 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.71 sec
2013-09-18 02:03:01,533 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.71 sec
2013-09-18 02:03:02,538 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.71 sec
2013-09-18 02:03:03,543 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 2.2 sec
2013-09-18 02:03:04,547 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 2.2 sec
2013-09-18 02:03:05,553 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 2.2 sec
MapReduce Total cumulative CPU time: 2 seconds 200 msec
Ended Job = job_201309172235_0182
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 11 Cumulative CPU: 474.77 sec HDFS Read: 988828092 HDFS Write: 1056 SUCCESS
Job 1: Map: 1 Reduce: 1 Cumulative CPU: 2.2 sec HDFS Read: 4633 HDFS Write: 0 SUCCESS
Total MapReduce CPU Time Spent: 7 minutes 56 seconds 970 msec
OK
Time taken: 67.483 seconds
hive> quit;
times: 2
query: SELECT URL, count(*) AS PageViews FROM hits_100m WHERE CounterID = 34 AND EventDate >= TIMESTAMP('2013-07-01') AND EventDate <= TIMESTAMP('2013-07-31') AND NOT Refresh != 0 AND IsLink != 0 AND NOT IsDownload != 0 GROUP BY URL ORDER BY PageViews DESC LIMIT 1000;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_10101@mturlrep13_201309180203_1414891537.txt
hive> SELECT URL, count(*) AS PageViews FROM hits_100m WHERE CounterID = 34 AND EventDate >= TIMESTAMP('2013-07-01') AND EventDate <= TIMESTAMP('2013-07-31') AND NOT Refresh != 0 AND IsLink != 0 AND NOT IsDownload != 0 GROUP BY URL ORDER BY PageViews DESC LIMIT 1000;;
Total MapReduce jobs = 2
Launching Job 1 out of 2
Number of reduce tasks not specified. Estimated from input data size: 11
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0183
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 11
2013-09-18 02:03:15,331 Stage-1 map = 0%, reduce = 0%
2013-09-18 02:03:25,419 Stage-1 map = 2%, reduce = 0%
2013-09-18 02:03:26,435 Stage-1 map = 10%, reduce = 0%
2013-09-18 02:03:28,457 Stage-1 map = 14%, reduce = 0%
2013-09-18 02:03:29,465 Stage-1 map = 37%, reduce = 0%
2013-09-18 02:03:30,480 Stage-1 map = 41%, reduce = 0%, Cumulative CPU 8.3 sec
2013-09-18 02:03:31,488 Stage-1 map = 44%, reduce = 0%, Cumulative CPU 8.3 sec
2013-09-18 02:03:32,496 Stage-1 map = 56%, reduce = 0%, Cumulative CPU 19.5 sec
2013-09-18 02:03:33,503 Stage-1 map = 63%, reduce = 0%, Cumulative CPU 31.5 sec
2013-09-18 02:03:34,511 Stage-1 map = 83%, reduce = 0%, Cumulative CPU 192.2 sec
2013-09-18 02:03:35,518 Stage-1 map = 99%, reduce = 0%, Cumulative CPU 435.25 sec
2013-09-18 02:03:36,525 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 463.51 sec
2013-09-18 02:03:37,531 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 463.51 sec
2013-09-18 02:03:38,536 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 463.51 sec
2013-09-18 02:03:39,542 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 463.51 sec
2013-09-18 02:03:40,548 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 463.51 sec
2013-09-18 02:03:41,554 Stage-1 map = 100%, reduce = 6%, Cumulative CPU 463.51 sec
2013-09-18 02:03:42,560 Stage-1 map = 100%, reduce = 24%, Cumulative CPU 463.51 sec
2013-09-18 02:03:43,568 Stage-1 map = 100%, reduce = 76%, Cumulative CPU 487.09 sec
2013-09-18 02:03:44,575 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 500.52 sec
2013-09-18 02:03:45,581 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 500.52 sec
2013-09-18 02:03:46,587 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 500.52 sec
MapReduce Total cumulative CPU time: 8 minutes 20 seconds 520 msec
Ended Job = job_201309172235_0183
Launching Job 2 out of 2
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0184
Hadoop job information for Stage-2: number of mappers: 1; number of reducers: 1
2013-09-18 02:03:49,037 Stage-2 map = 0%, reduce = 0%
2013-09-18 02:03:51,046 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.71 sec
2013-09-18 02:03:52,051 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.71 sec
2013-09-18 02:03:53,056 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.71 sec
2013-09-18 02:03:54,061 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.71 sec
2013-09-18 02:03:55,066 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.71 sec
2013-09-18 02:03:56,071 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.71 sec
2013-09-18 02:03:57,076 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.71 sec
2013-09-18 02:03:58,080 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.71 sec
2013-09-18 02:03:59,086 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 2.34 sec
2013-09-18 02:04:00,091 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 2.34 sec
2013-09-18 02:04:01,096 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 2.34 sec
MapReduce Total cumulative CPU time: 2 seconds 340 msec
Ended Job = job_201309172235_0184
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 11 Cumulative CPU: 500.52 sec HDFS Read: 988828092 HDFS Write: 1056 SUCCESS
Job 1: Map: 1 Reduce: 1 Cumulative CPU: 2.34 sec HDFS Read: 4633 HDFS Write: 0 SUCCESS
Total MapReduce CPU Time Spent: 8 minutes 22 seconds 860 msec
OK
Time taken: 53.704 seconds
hive> quit;
times: 3
query: SELECT URL, count(*) AS PageViews FROM hits_100m WHERE CounterID = 34 AND EventDate >= TIMESTAMP('2013-07-01') AND EventDate <= TIMESTAMP('2013-07-31') AND NOT Refresh != 0 AND IsLink != 0 AND NOT IsDownload != 0 GROUP BY URL ORDER BY PageViews DESC LIMIT 1000;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_17498@mturlrep13_201309180204_309695720.txt
hive> SELECT URL, count(*) AS PageViews FROM hits_100m WHERE CounterID = 34 AND EventDate >= TIMESTAMP('2013-07-01') AND EventDate <= TIMESTAMP('2013-07-31') AND NOT Refresh != 0 AND IsLink != 0 AND NOT IsDownload != 0 GROUP BY URL ORDER BY PageViews DESC LIMIT 1000;;
Total MapReduce jobs = 2
Launching Job 1 out of 2
Number of reduce tasks not specified. Estimated from input data size: 11
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0185
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 11
2013-09-18 02:04:11,776 Stage-1 map = 0%, reduce = 0%
2013-09-18 02:04:20,938 Stage-1 map = 3%, reduce = 0%
2013-09-18 02:04:21,952 Stage-1 map = 11%, reduce = 0%
2013-09-18 02:04:23,979 Stage-1 map = 13%, reduce = 0%
2013-09-18 02:04:24,984 Stage-1 map = 37%, reduce = 0%
2013-09-18 02:04:25,990 Stage-1 map = 44%, reduce = 0%
2013-09-18 02:04:28,019 Stage-1 map = 53%, reduce = 0%, Cumulative CPU 8.73 sec
2013-09-18 02:04:29,048 Stage-1 map = 68%, reduce = 0%, Cumulative CPU 43.13 sec
2013-09-18 02:04:30,071 Stage-1 map = 84%, reduce = 0%, Cumulative CPU 227.84 sec
2013-09-18 02:04:31,079 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 459.59 sec
2013-09-18 02:04:32,089 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 459.59 sec
2013-09-18 02:04:33,095 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 459.59 sec
2013-09-18 02:04:34,101 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 459.59 sec
2013-09-18 02:04:35,106 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 459.59 sec
2013-09-18 02:04:36,112 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 459.59 sec
2013-09-18 02:04:37,119 Stage-1 map = 100%, reduce = 3%, Cumulative CPU 459.59 sec
2013-09-18 02:04:38,128 Stage-1 map = 100%, reduce = 27%, Cumulative CPU 462.59 sec
2013-09-18 02:04:39,135 Stage-1 map = 100%, reduce = 61%, Cumulative CPU 476.43 sec
2013-09-18 02:04:40,142 Stage-1 map = 100%, reduce = 88%, Cumulative CPU 490.11 sec
2013-09-18 02:04:41,149 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 496.41 sec
2013-09-18 02:04:42,156 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 496.41 sec
2013-09-18 02:04:43,162 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 496.41 sec
MapReduce Total cumulative CPU time: 8 minutes 16 seconds 410 msec
Ended Job = job_201309172235_0185
Launching Job 2 out of 2
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0186
Hadoop job information for Stage-2: number of mappers: 1; number of reducers: 1
2013-09-18 02:04:45,666 Stage-2 map = 0%, reduce = 0%
2013-09-18 02:04:47,674 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.7 sec
2013-09-18 02:04:48,680 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.7 sec
2013-09-18 02:04:49,685 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.7 sec
2013-09-18 02:04:50,689 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.7 sec
2013-09-18 02:04:51,694 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.7 sec
2013-09-18 02:04:52,699 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.7 sec
2013-09-18 02:04:53,703 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.7 sec
2013-09-18 02:04:54,707 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 0.7 sec
2013-09-18 02:04:55,712 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 2.15 sec
2013-09-18 02:04:56,718 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 2.15 sec
2013-09-18 02:04:57,723 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 2.15 sec
MapReduce Total cumulative CPU time: 2 seconds 150 msec
Ended Job = job_201309172235_0186
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 11 Cumulative CPU: 496.41 sec HDFS Read: 988828092 HDFS Write: 1056 SUCCESS
Job 1: Map: 1 Reduce: 1 Cumulative CPU: 2.15 sec HDFS Read: 4633 HDFS Write: 0 SUCCESS
Total MapReduce CPU Time Spent: 8 minutes 18 seconds 560 msec
OK
Time taken: 54.814 seconds
hive> quit;
times: 1
query: SELECT TraficSourceID, SearchEngineID, AdvEngineID, URL, count(*), if(SearchEngineID = 0 AND AdvEngineID = 0 , Referer, '') AS SRC FROM hits_100m WHERE CounterID = 34 AND EventDate >= TIMESTAMP('2013-07-01') AND EventDate <= TIMESTAMP('2013-07-31') AND NOT Refresh != 0 GROUP BY TraficSourceID, SearchEngineID, AdvEngineID, if(SearchEngineID = 0 AND AdvEngineID = 0 , Referer, ''), URL ORDER BY count(*) DESC LIMIT 1000;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_24933@mturlrep13_201309180205_1377920242.txt
hive> SELECT TraficSourceID, SearchEngineID, AdvEngineID, URL, count(*), if(SearchEngineID = 0 AND AdvEngineID = 0 , Referer, '') AS SRC FROM hits_100m WHERE CounterID = 34 AND EventDate >= TIMESTAMP('2013-07-01') AND EventDate <= TIMESTAMP('2013-07-31') AND NOT Refresh != 0 GROUP BY TraficSourceID, SearchEngineID, AdvEngineID, if(SearchEngineID = 0 AND AdvEngineID = 0 , Referer, ''), URL ORDER BY count(*) DESC LIMIT 1000; ;
FAILED: SemanticException [Error 10128]: Line 1:415 Not yet supported place for UDAF 'count'
hive> quit;
times: 2
query: SELECT TraficSourceID, SearchEngineID, AdvEngineID, URL, count(*), if(SearchEngineID = 0 AND AdvEngineID = 0 , Referer, '') AS SRC FROM hits_100m WHERE CounterID = 34 AND EventDate >= TIMESTAMP('2013-07-01') AND EventDate <= TIMESTAMP('2013-07-31') AND NOT Refresh != 0 GROUP BY TraficSourceID, SearchEngineID, AdvEngineID, if(SearchEngineID = 0 AND AdvEngineID = 0 , Referer, ''), URL ORDER BY count(*) DESC LIMIT 1000;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_25460@mturlrep13_201309180205_1132498226.txt
hive> SELECT TraficSourceID, SearchEngineID, AdvEngineID, URL, count(*), if(SearchEngineID = 0 AND AdvEngineID = 0 , Referer, '') AS SRC FROM hits_100m WHERE CounterID = 34 AND EventDate >= TIMESTAMP('2013-07-01') AND EventDate <= TIMESTAMP('2013-07-31') AND NOT Refresh != 0 GROUP BY TraficSourceID, SearchEngineID, AdvEngineID, if(SearchEngineID = 0 AND AdvEngineID = 0 , Referer, ''), URL ORDER BY count(*) DESC LIMIT 1000; ;
FAILED: SemanticException [Error 10128]: Line 1:415 Not yet supported place for UDAF 'count'
hive> quit;
times: 3
query: SELECT TraficSourceID, SearchEngineID, AdvEngineID, URL, count(*), if(SearchEngineID = 0 AND AdvEngineID = 0 , Referer, '') AS SRC FROM hits_100m WHERE CounterID = 34 AND EventDate >= TIMESTAMP('2013-07-01') AND EventDate <= TIMESTAMP('2013-07-31') AND NOT Refresh != 0 GROUP BY TraficSourceID, SearchEngineID, AdvEngineID, if(SearchEngineID = 0 AND AdvEngineID = 0 , Referer, ''), URL ORDER BY count(*) DESC LIMIT 1000;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_26035@mturlrep13_201309180205_978107887.txt
hive> SELECT TraficSourceID, SearchEngineID, AdvEngineID, URL, count(*), if(SearchEngineID = 0 AND AdvEngineID = 0 , Referer, '') AS SRC FROM hits_100m WHERE CounterID = 34 AND EventDate >= TIMESTAMP('2013-07-01') AND EventDate <= TIMESTAMP('2013-07-31') AND NOT Refresh != 0 GROUP BY TraficSourceID, SearchEngineID, AdvEngineID, if(SearchEngineID = 0 AND AdvEngineID = 0 , Referer, ''), URL ORDER BY count(*) DESC LIMIT 1000; ;
FAILED: SemanticException [Error 10128]: Line 1:415 Not yet supported place for UDAF 'count'
hive> quit;
times: 1
query: SELECT URLHash, EventDate, count(*) AS PageViews FROM hits_100m WHERE CounterID = 34 AND EventDate >= TIMESTAMP('2013-07-01') AND EventDate <= TIMESTAMP('2013-07-31') AND NOT Refresh != 0 AND TraficSourceID IN (-1, 6) AND RefererHash = 6202628419148573758 GROUP BY URLHash, EventDate ORDER BY PageViews DESC LIMIT 100000;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_26250@mturlrep13_201309180205_637154344.txt
hive> SELECT URLHash, EventDate, count(*) AS PageViews FROM hits_100m WHERE CounterID = 34 AND EventDate >= TIMESTAMP('2013-07-01') AND EventDate <= TIMESTAMP('2013-07-31') AND NOT Refresh != 0 AND TraficSourceID IN (-1, 6) AND RefererHash = 6202628419148573758 GROUP BY URLHash, EventDate ORDER BY PageViews DESC LIMIT 100000; ;
Total MapReduce jobs = 2
Launching Job 1 out of 2
Number of reduce tasks not specified. Estimated from input data size: 11
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0187
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 11
2013-09-18 02:05:41,227 Stage-1 map = 0%, reduce = 0%
2013-09-18 02:05:51,363 Stage-1 map = 5%, reduce = 0%
2013-09-18 02:05:54,380 Stage-1 map = 16%, reduce = 0%
2013-09-18 02:05:57,395 Stage-1 map = 27%, reduce = 0%
2013-09-18 02:05:58,400 Stage-1 map = 30%, reduce = 0%
2013-09-18 02:06:00,410 Stage-1 map = 42%, reduce = 0%
2013-09-18 02:06:01,424 Stage-1 map = 47%, reduce = 0%, Cumulative CPU 17.15 sec
2013-09-18 02:06:02,445 Stage-1 map = 47%, reduce = 0%, Cumulative CPU 262.15 sec
2013-09-18 02:06:04,340 Stage-1 map = 47%, reduce = 0%, Cumulative CPU 262.15 sec
2013-09-18 02:06:05,365 Stage-1 map = 59%, reduce = 0%, Cumulative CPU 262.15 sec
2013-09-18 02:06:06,379 Stage-1 map = 68%, reduce = 0%, Cumulative CPU 285.59 sec
2013-09-18 02:06:07,390 Stage-1 map = 71%, reduce = 0%, Cumulative CPU 291.15 sec
2013-09-18 02:06:08,397 Stage-1 map = 90%, reduce = 0%, Cumulative CPU 320.44 sec
2013-09-18 02:06:09,404 Stage-1 map = 97%, reduce = 0%, Cumulative CPU 364.61 sec
2013-09-18 02:06:10,410 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 373.66 sec
2013-09-18 02:06:11,417 Stage-1 map = 100%, reduce = 3%, Cumulative CPU 373.66 sec
2013-09-18 02:06:12,424 Stage-1 map = 100%, reduce = 12%, Cumulative CPU 373.66 sec
2013-09-18 02:06:13,433 Stage-1 map = 100%, reduce = 36%, Cumulative CPU 379.8 sec
2013-09-18 02:06:14,439 Stage-1 map = 100%, reduce = 73%, Cumulative CPU 396.64 sec
2013-09-18 02:06:15,446 Stage-1 map = 100%, reduce = 94%, Cumulative CPU 407.2 sec
2013-09-18 02:06:16,453 Stage-1 map = 100%, reduce = 94%, Cumulative CPU 407.2 sec
2013-09-18 02:06:17,460 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 410.44 sec
2013-09-18 02:06:18,468 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 410.44 sec
MapReduce Total cumulative CPU time: 6 minutes 50 seconds 440 msec
Ended Job = job_201309172235_0187
Launching Job 2 out of 2
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0188
Hadoop job information for Stage-2: number of mappers: 1; number of reducers: 1
2013-09-18 02:06:21,966 Stage-2 map = 0%, reduce = 0%
2013-09-18 02:06:22,972 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.79 sec
2013-09-18 02:06:23,978 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.79 sec
2013-09-18 02:06:24,984 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.79 sec
2013-09-18 02:06:25,990 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.79 sec
2013-09-18 02:06:26,994 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.79 sec
2013-09-18 02:06:27,999 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.79 sec
2013-09-18 02:06:29,004 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.79 sec
2013-09-18 02:06:30,008 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.79 sec
2013-09-18 02:06:31,015 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 2.41 sec
2013-09-18 02:06:32,021 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 2.41 sec
2013-09-18 02:06:33,027 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 2.41 sec
MapReduce Total cumulative CPU time: 2 seconds 410 msec
Ended Job = job_201309172235_0188
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 11 Cumulative CPU: 410.44 sec HDFS Read: 1326437326 HDFS Write: 1056 SUCCESS
Job 1: Map: 1 Reduce: 1 Cumulative CPU: 2.41 sec HDFS Read: 4633 HDFS Write: 0 SUCCESS
Total MapReduce CPU Time Spent: 6 minutes 52 seconds 850 msec
OK
Time taken: 62.644 seconds
hive> quit;
times: 2
query: SELECT URLHash, EventDate, count(*) AS PageViews FROM hits_100m WHERE CounterID = 34 AND EventDate >= TIMESTAMP('2013-07-01') AND EventDate <= TIMESTAMP('2013-07-31') AND NOT Refresh != 0 AND TraficSourceID IN (-1, 6) AND RefererHash = 6202628419148573758 GROUP BY URLHash, EventDate ORDER BY PageViews DESC LIMIT 100000;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_1255@mturlrep13_201309180206_866108725.txt
hive> SELECT URLHash, EventDate, count(*) AS PageViews FROM hits_100m WHERE CounterID = 34 AND EventDate >= TIMESTAMP('2013-07-01') AND EventDate <= TIMESTAMP('2013-07-31') AND NOT Refresh != 0 AND TraficSourceID IN (-1, 6) AND RefererHash = 6202628419148573758 GROUP BY URLHash, EventDate ORDER BY PageViews DESC LIMIT 100000; ;
Total MapReduce jobs = 2
Launching Job 1 out of 2
Number of reduce tasks not specified. Estimated from input data size: 11
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0189
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 11
2013-09-18 02:06:42,810 Stage-1 map = 0%, reduce = 0%
2013-09-18 02:06:53,890 Stage-1 map = 1%, reduce = 0%
2013-09-18 02:06:54,899 Stage-1 map = 3%, reduce = 0%
2013-09-18 02:06:55,912 Stage-1 map = 12%, reduce = 0%
2013-09-18 02:06:56,943 Stage-1 map = 19%, reduce = 0%
2013-09-18 02:06:57,951 Stage-1 map = 20%, reduce = 0%
2013-09-18 02:06:58,970 Stage-1 map = 55%, reduce = 0%, Cumulative CPU 9.16 sec
2013-09-18 02:07:00,000 Stage-1 map = 66%, reduce = 0%, Cumulative CPU 83.33 sec
2013-09-18 02:07:01,028 Stage-1 map = 78%, reduce = 0%, Cumulative CPU 218.77 sec
2013-09-18 02:07:02,042 Stage-1 map = 99%, reduce = 0%, Cumulative CPU 399.52 sec
2013-09-18 02:07:03,049 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 425.85 sec
2013-09-18 02:07:04,055 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 425.85 sec
2013-09-18 02:07:05,061 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 425.85 sec
2013-09-18 02:07:06,067 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 425.85 sec
2013-09-18 02:07:07,073 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 425.85 sec
2013-09-18 02:07:08,080 Stage-1 map = 100%, reduce = 9%, Cumulative CPU 425.85 sec
2013-09-18 02:07:09,086 Stage-1 map = 100%, reduce = 21%, Cumulative CPU 425.85 sec
2013-09-18 02:07:10,096 Stage-1 map = 100%, reduce = 70%, Cumulative CPU 446.81 sec
2013-09-18 02:07:11,102 Stage-1 map = 100%, reduce = 94%, Cumulative CPU 459.98 sec
2013-09-18 02:07:12,109 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 463.17 sec
2013-09-18 02:07:13,117 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 463.17 sec
MapReduce Total cumulative CPU time: 7 minutes 43 seconds 170 msec
Ended Job = job_201309172235_0189
Launching Job 2 out of 2
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0190
Hadoop job information for Stage-2: number of mappers: 1; number of reducers: 1
2013-09-18 02:07:15,611 Stage-2 map = 0%, reduce = 0%
2013-09-18 02:07:17,620 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.73 sec
2013-09-18 02:07:18,626 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.73 sec
2013-09-18 02:07:19,631 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.73 sec
2013-09-18 02:07:20,636 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.73 sec
2013-09-18 02:07:21,641 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.73 sec
2013-09-18 02:07:22,650 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.73 sec
2013-09-18 02:07:23,655 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.73 sec
2013-09-18 02:07:24,660 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 0.73 sec
2013-09-18 02:07:25,666 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 2.32 sec
2013-09-18 02:07:26,675 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 2.32 sec
MapReduce Total cumulative CPU time: 2 seconds 320 msec
Ended Job = job_201309172235_0190
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 11 Cumulative CPU: 463.17 sec HDFS Read: 1326437326 HDFS Write: 1056 SUCCESS
Job 1: Map: 1 Reduce: 1 Cumulative CPU: 2.32 sec HDFS Read: 4633 HDFS Write: 0 SUCCESS
Total MapReduce CPU Time Spent: 7 minutes 45 seconds 490 msec
OK
Time taken: 51.812 seconds
hive> quit;
times: 3
query: SELECT URLHash, EventDate, count(*) AS PageViews FROM hits_100m WHERE CounterID = 34 AND EventDate >= TIMESTAMP('2013-07-01') AND EventDate <= TIMESTAMP('2013-07-31') AND NOT Refresh != 0 AND TraficSourceID IN (-1, 6) AND RefererHash = 6202628419148573758 GROUP BY URLHash, EventDate ORDER BY PageViews DESC LIMIT 100000;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_9097@mturlrep13_201309180207_752390467.txt
hive> SELECT URLHash, EventDate, count(*) AS PageViews FROM hits_100m WHERE CounterID = 34 AND EventDate >= TIMESTAMP('2013-07-01') AND EventDate <= TIMESTAMP('2013-07-31') AND NOT Refresh != 0 AND TraficSourceID IN (-1, 6) AND RefererHash = 6202628419148573758 GROUP BY URLHash, EventDate ORDER BY PageViews DESC LIMIT 100000; ;
Total MapReduce jobs = 2
Launching Job 1 out of 2
Number of reduce tasks not specified. Estimated from input data size: 11
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0191
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 11
2013-09-18 02:07:36,443 Stage-1 map = 0%, reduce = 0%
2013-09-18 02:07:48,547 Stage-1 map = 3%, reduce = 0%
2013-09-18 02:07:49,583 Stage-1 map = 13%, reduce = 0%
2013-09-18 02:07:50,619 Stage-1 map = 15%, reduce = 0%
2013-09-18 02:07:51,643 Stage-1 map = 22%, reduce = 0%
2013-09-18 02:07:52,690 Stage-1 map = 54%, reduce = 0%, Cumulative CPU 9.35 sec
2013-09-18 02:07:53,721 Stage-1 map = 62%, reduce = 0%, Cumulative CPU 50.99 sec
2013-09-18 02:07:54,737 Stage-1 map = 81%, reduce = 0%, Cumulative CPU 242.49 sec
2013-09-18 02:07:55,745 Stage-1 map = 99%, reduce = 0%, Cumulative CPU 412.77 sec
2013-09-18 02:07:56,751 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 426.23 sec
2013-09-18 02:07:57,758 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 426.23 sec
2013-09-18 02:07:58,763 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 426.23 sec
2013-09-18 02:07:59,770 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 426.23 sec
2013-09-18 02:08:00,776 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 426.23 sec
2013-09-18 02:08:01,781 Stage-1 map = 100%, reduce = 9%, Cumulative CPU 426.23 sec
2013-09-18 02:08:02,791 Stage-1 map = 100%, reduce = 27%, Cumulative CPU 433.68 sec
2013-09-18 02:08:03,799 Stage-1 map = 100%, reduce = 61%, Cumulative CPU 444.3 sec
2013-09-18 02:08:04,806 Stage-1 map = 100%, reduce = 94%, Cumulative CPU 460.63 sec
2013-09-18 02:08:05,812 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 464.02 sec
2013-09-18 02:08:06,819 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 464.02 sec
MapReduce Total cumulative CPU time: 7 minutes 44 seconds 20 msec
Ended Job = job_201309172235_0191
Launching Job 2 out of 2
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0192
Hadoop job information for Stage-2: number of mappers: 1; number of reducers: 1
2013-09-18 02:08:09,707 Stage-2 map = 0%, reduce = 0%
2013-09-18 02:08:11,717 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.74 sec
2013-09-18 02:08:12,723 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.74 sec
2013-09-18 02:08:13,729 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.74 sec
2013-09-18 02:08:14,734 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.74 sec
2013-09-18 02:08:15,740 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.74 sec
2013-09-18 02:08:16,744 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.74 sec
2013-09-18 02:08:17,749 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.74 sec
2013-09-18 02:08:18,754 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 0.74 sec
2013-09-18 02:08:19,760 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 2.29 sec
2013-09-18 02:08:20,766 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 2.29 sec
2013-09-18 02:08:21,771 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 2.29 sec
MapReduce Total cumulative CPU time: 2 seconds 290 msec
Ended Job = job_201309172235_0192
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 11 Cumulative CPU: 464.02 sec HDFS Read: 1326437326 HDFS Write: 1056 SUCCESS
Job 1: Map: 1 Reduce: 1 Cumulative CPU: 2.29 sec HDFS Read: 4622 HDFS Write: 0 SUCCESS
Total MapReduce CPU Time Spent: 7 minutes 46 seconds 310 msec
OK
Time taken: 53.23 seconds
hive> quit;
times: 1
query: SELECT WindowClientWidth, WindowClientHeight, count(*) AS PageViews FROM hits_100m WHERE CounterID = 34 AND EventDate >= TIMESTAMP('2013-07-01') AND EventDate <= TIMESTAMP('2013-07-31') AND NOT Refresh != 0 AND NOT DontCountHits != 0 AND URLHash = 6202628419148573758 GROUP BY WindowClientWidth, WindowClientHeight ORDER BY PageViews DESC LIMIT 10000;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_16494@mturlrep13_201309180208_1768463196.txt
hive> SELECT WindowClientWidth, WindowClientHeight, count(*) AS PageViews FROM hits_100m WHERE CounterID = 34 AND EventDate >= TIMESTAMP('2013-07-01') AND EventDate <= TIMESTAMP('2013-07-31') AND NOT Refresh != 0 AND NOT DontCountHits != 0 AND URLHash = 6202628419148573758 GROUP BY WindowClientWidth, WindowClientHeight ORDER BY PageViews DESC LIMIT 10000; ;
Total MapReduce jobs = 2
Launching Job 1 out of 2
Number of reduce tasks not specified. Estimated from input data size: 11
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0193
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 11
2013-09-18 02:08:39,476 Stage-1 map = 0%, reduce = 0%
2013-09-18 02:08:52,587 Stage-1 map = 2%, reduce = 0%
2013-09-18 02:08:53,593 Stage-1 map = 3%, reduce = 0%
2013-09-18 02:08:58,633 Stage-1 map = 18%, reduce = 0%
2013-09-18 02:08:59,640 Stage-1 map = 22%, reduce = 0%
2013-09-18 02:09:01,649 Stage-1 map = 37%, reduce = 0%
2013-09-18 02:09:02,654 Stage-1 map = 38%, reduce = 0%
2013-09-18 02:09:04,909 Stage-1 map = 39%, reduce = 0%, Cumulative CPU 242.14 sec
2013-09-18 02:09:05,923 Stage-1 map = 49%, reduce = 0%, Cumulative CPU 243.18 sec
2013-09-18 02:09:06,929 Stage-1 map = 49%, reduce = 0%, Cumulative CPU 243.18 sec
2013-09-18 02:09:07,941 Stage-1 map = 50%, reduce = 0%, Cumulative CPU 244.77 sec
2013-09-18 02:09:08,962 Stage-1 map = 50%, reduce = 0%, Cumulative CPU 244.77 sec
2013-09-18 02:09:10,398 Stage-1 map = 66%, reduce = 0%, Cumulative CPU 250.0 sec
2013-09-18 02:09:11,404 Stage-1 map = 76%, reduce = 0%, Cumulative CPU 278.26 sec
2013-09-18 02:09:12,413 Stage-1 map = 76%, reduce = 0%, Cumulative CPU 278.26 sec
2013-09-18 02:09:13,428 Stage-1 map = 83%, reduce = 0%, Cumulative CPU 289.83 sec
2013-09-18 02:09:14,435 Stage-1 map = 97%, reduce = 1%, Cumulative CPU 362.43 sec
2013-09-18 02:09:15,442 Stage-1 map = 100%, reduce = 1%, Cumulative CPU 399.93 sec
2013-09-18 02:09:16,449 Stage-1 map = 100%, reduce = 4%, Cumulative CPU 399.93 sec
2013-09-18 02:09:17,455 Stage-1 map = 100%, reduce = 15%, Cumulative CPU 399.93 sec
2013-09-18 02:09:18,463 Stage-1 map = 100%, reduce = 45%, Cumulative CPU 412.86 sec
2013-09-18 02:09:19,469 Stage-1 map = 100%, reduce = 63%, Cumulative CPU 419.48 sec
2013-09-18 02:09:20,476 Stage-1 map = 100%, reduce = 70%, Cumulative CPU 419.48 sec
2013-09-18 02:09:21,483 Stage-1 map = 100%, reduce = 94%, Cumulative CPU 434.29 sec
2013-09-18 02:09:22,490 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 437.36 sec
2013-09-18 02:09:23,496 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 437.36 sec
MapReduce Total cumulative CPU time: 7 minutes 17 seconds 360 msec
Ended Job = job_201309172235_0193
Launching Job 2 out of 2
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0194
Hadoop job information for Stage-2: number of mappers: 1; number of reducers: 1
2013-09-18 02:09:27,006 Stage-2 map = 0%, reduce = 0%
2013-09-18 02:09:28,011 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.7 sec
2013-09-18 02:09:29,015 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.7 sec
2013-09-18 02:09:30,021 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.7 sec
2013-09-18 02:09:31,024 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.7 sec
2013-09-18 02:09:32,028 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.7 sec
2013-09-18 02:09:33,032 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.7 sec
2013-09-18 02:09:34,037 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.7 sec
2013-09-18 02:09:35,041 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.7 sec
2013-09-18 02:09:36,047 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 2.21 sec
2013-09-18 02:09:37,052 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 2.21 sec
2013-09-18 02:09:38,057 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 2.21 sec
MapReduce Total cumulative CPU time: 2 seconds 210 msec
Ended Job = job_201309172235_0194
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 11 Cumulative CPU: 437.36 sec HDFS Read: 1004716597 HDFS Write: 1056 SUCCESS
Job 1: Map: 1 Reduce: 1 Cumulative CPU: 2.21 sec HDFS Read: 4633 HDFS Write: 0 SUCCESS
Total MapReduce CPU Time Spent: 7 minutes 19 seconds 570 msec
OK
Time taken: 69.068 seconds
hive> quit;
times: 2
query: SELECT WindowClientWidth, WindowClientHeight, count(*) AS PageViews FROM hits_100m WHERE CounterID = 34 AND EventDate >= TIMESTAMP('2013-07-01') AND EventDate <= TIMESTAMP('2013-07-31') AND NOT Refresh != 0 AND NOT DontCountHits != 0 AND URLHash = 6202628419148573758 GROUP BY WindowClientWidth, WindowClientHeight ORDER BY PageViews DESC LIMIT 10000;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_24195@mturlrep13_201309180209_485949400.txt
hive> SELECT WindowClientWidth, WindowClientHeight, count(*) AS PageViews FROM hits_100m WHERE CounterID = 34 AND EventDate >= TIMESTAMP('2013-07-01') AND EventDate <= TIMESTAMP('2013-07-31') AND NOT Refresh != 0 AND NOT DontCountHits != 0 AND URLHash = 6202628419148573758 GROUP BY WindowClientWidth, WindowClientHeight ORDER BY PageViews DESC LIMIT 10000; ;
Total MapReduce jobs = 2
Launching Job 1 out of 2
Number of reduce tasks not specified. Estimated from input data size: 11
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0195
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 11
2013-09-18 02:09:47,707 Stage-1 map = 0%, reduce = 0%
2013-09-18 02:09:58,784 Stage-1 map = 1%, reduce = 0%
2013-09-18 02:09:59,803 Stage-1 map = 8%, reduce = 0%
2013-09-18 02:10:00,817 Stage-1 map = 13%, reduce = 0%
2013-09-18 02:10:01,836 Stage-1 map = 18%, reduce = 0%
2013-09-18 02:10:02,908 Stage-1 map = 32%, reduce = 0%, Cumulative CPU 279.26 sec
2013-09-18 02:10:03,925 Stage-1 map = 55%, reduce = 0%, Cumulative CPU 286.13 sec
2013-09-18 02:10:04,949 Stage-1 map = 62%, reduce = 0%, Cumulative CPU 292.13 sec
2013-09-18 02:10:05,955 Stage-1 map = 83%, reduce = 0%, Cumulative CPU 359.01 sec
2013-09-18 02:10:06,963 Stage-1 map = 99%, reduce = 0%, Cumulative CPU 409.34 sec
2013-09-18 02:10:07,970 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 422.22 sec
2013-09-18 02:10:08,976 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 422.22 sec
2013-09-18 02:10:09,981 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 422.22 sec
2013-09-18 02:10:10,987 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 422.22 sec
2013-09-18 02:10:11,993 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 422.22 sec
2013-09-18 02:10:13,002 Stage-1 map = 100%, reduce = 12%, Cumulative CPU 422.22 sec
2013-09-18 02:10:14,012 Stage-1 map = 100%, reduce = 30%, Cumulative CPU 425.38 sec
2013-09-18 02:10:15,028 Stage-1 map = 100%, reduce = 76%, Cumulative CPU 448.26 sec
2013-09-18 02:10:16,035 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 462.58 sec
2013-09-18 02:10:17,041 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 462.58 sec
2013-09-18 02:10:18,047 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 462.58 sec
MapReduce Total cumulative CPU time: 7 minutes 42 seconds 580 msec
Ended Job = job_201309172235_0195
Launching Job 2 out of 2
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0196
Hadoop job information for Stage-2: number of mappers: 1; number of reducers: 1
2013-09-18 02:10:22,087 Stage-2 map = 0%, reduce = 0%
2013-09-18 02:10:24,096 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.78 sec
2013-09-18 02:10:25,101 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.78 sec
2013-09-18 02:10:26,106 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.78 sec
2013-09-18 02:10:27,111 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.78 sec
2013-09-18 02:10:28,116 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.78 sec
2013-09-18 02:10:29,121 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.78 sec
2013-09-18 02:10:30,126 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.78 sec
2013-09-18 02:10:31,131 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 2.25 sec
2013-09-18 02:10:32,137 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 2.25 sec
2013-09-18 02:10:33,142 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 2.25 sec
MapReduce Total cumulative CPU time: 2 seconds 250 msec
Ended Job = job_201309172235_0196
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 11 Cumulative CPU: 462.58 sec HDFS Read: 1004716597 HDFS Write: 1056 SUCCESS
Job 1: Map: 1 Reduce: 1 Cumulative CPU: 2.25 sec HDFS Read: 4633 HDFS Write: 0 SUCCESS
Total MapReduce CPU Time Spent: 7 minutes 44 seconds 830 msec
OK
Time taken: 53.234 seconds
hive> quit;
times: 3
query: SELECT WindowClientWidth, WindowClientHeight, count(*) AS PageViews FROM hits_100m WHERE CounterID = 34 AND EventDate >= TIMESTAMP('2013-07-01') AND EventDate <= TIMESTAMP('2013-07-31') AND NOT Refresh != 0 AND NOT DontCountHits != 0 AND URLHash = 6202628419148573758 GROUP BY WindowClientWidth, WindowClientHeight ORDER BY PageViews DESC LIMIT 10000;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_32545@mturlrep13_201309180210_1142487279.txt
hive> SELECT WindowClientWidth, WindowClientHeight, count(*) AS PageViews FROM hits_100m WHERE CounterID = 34 AND EventDate >= TIMESTAMP('2013-07-01') AND EventDate <= TIMESTAMP('2013-07-31') AND NOT Refresh != 0 AND NOT DontCountHits != 0 AND URLHash = 6202628419148573758 GROUP BY WindowClientWidth, WindowClientHeight ORDER BY PageViews DESC LIMIT 10000; ;
Total MapReduce jobs = 2
Launching Job 1 out of 2
Number of reduce tasks not specified. Estimated from input data size: 11
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0197
Hadoop job information for Stage-1: number of mappers: 37; number of reducers: 11
2013-09-18 02:10:43,786 Stage-1 map = 0%, reduce = 0%
2013-09-18 02:10:53,999 Stage-1 map = 1%, reduce = 0%
2013-09-18 02:10:55,007 Stage-1 map = 4%, reduce = 0%
2013-09-18 02:10:56,013 Stage-1 map = 11%, reduce = 0%
2013-09-18 02:10:57,019 Stage-1 map = 17%, reduce = 0%
2013-09-18 02:10:58,030 Stage-1 map = 28%, reduce = 0%
2013-09-18 02:10:59,047 Stage-1 map = 48%, reduce = 0%
2013-09-18 02:11:00,066 Stage-1 map = 61%, reduce = 0%, Cumulative CPU 51.25 sec
2013-09-18 02:11:01,076 Stage-1 map = 78%, reduce = 0%, Cumulative CPU 185.97 sec
2013-09-18 02:11:02,087 Stage-1 map = 99%, reduce = 0%, Cumulative CPU 391.71 sec
2013-09-18 02:11:03,100 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 432.05 sec
2013-09-18 02:11:04,107 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 432.05 sec
2013-09-18 02:11:05,113 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 432.05 sec
2013-09-18 02:11:06,118 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 432.05 sec
2013-09-18 02:11:07,124 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 432.05 sec
2013-09-18 02:11:08,130 Stage-1 map = 100%, reduce = 6%, Cumulative CPU 432.05 sec
2013-09-18 02:11:09,140 Stage-1 map = 100%, reduce = 36%, Cumulative CPU 438.09 sec
2013-09-18 02:11:10,147 Stage-1 map = 100%, reduce = 70%, Cumulative CPU 452.13 sec
2013-09-18 02:11:11,154 Stage-1 map = 100%, reduce = 94%, Cumulative CPU 466.84 sec
2013-09-18 02:11:12,160 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 470.1 sec
2013-09-18 02:11:13,166 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 470.1 sec
MapReduce Total cumulative CPU time: 7 minutes 50 seconds 100 msec
Ended Job = job_201309172235_0197
Launching Job 2 out of 2
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Kill Command = /usr/libexec/../bin/hadoop job -kill job_201309172235_0198
Hadoop job information for Stage-2: number of mappers: 1; number of reducers: 1
2013-09-18 02:11:16,669 Stage-2 map = 0%, reduce = 0%
2013-09-18 02:11:17,675 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.73 sec
2013-09-18 02:11:18,680 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.73 sec
2013-09-18 02:11:19,686 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.73 sec
2013-09-18 02:11:20,691 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.73 sec
2013-09-18 02:11:21,696 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.73 sec
2013-09-18 02:11:22,700 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.73 sec
2013-09-18 02:11:23,705 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.73 sec
2013-09-18 02:11:24,711 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 0.73 sec
2013-09-18 02:11:25,716 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 2.26 sec
2013-09-18 02:11:26,721 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 2.26 sec
2013-09-18 02:11:27,727 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 2.26 sec
MapReduce Total cumulative CPU time: 2 seconds 260 msec
Ended Job = job_201309172235_0198
MapReduce Jobs Launched:
Job 0: Map: 37 Reduce: 11 Cumulative CPU: 470.1 sec HDFS Read: 1004716597 HDFS Write: 1056 SUCCESS
Job 1: Map: 1 Reduce: 1 Cumulative CPU: 2.26 sec HDFS Read: 4633 HDFS Write: 0 SUCCESS
Total MapReduce CPU Time Spent: 7 minutes 52 seconds 360 msec
OK
Time taken: 52.853 seconds
hive> quit;
times: 1
query: SELECT unix_timestamp(EventTime) - SECOND(EventTime), count(*) FROM hits_100m WHERE CounterID = 34 AND EventDate >= TIMESTAMP('2013-07-01') AND EventDate <= TIMESTAMP('2013-07-02') AND NOT Refresh != 0 AND NOT DontCountHits != 0 GROUP BY m ORDER BY m;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_8277@mturlrep13_201309180211_1471622.txt
hive> SELECT unix_timestamp(EventTime) - SECOND(EventTime), count(*) FROM hits_100m WHERE CounterID = 34 AND EventDate >= TIMESTAMP('2013-07-01') AND EventDate <= TIMESTAMP('2013-07-02') AND NOT Refresh != 0 AND NOT DontCountHits != 0 GROUP BY m ORDER BY m; ;
hive> quit;
times: 2
query: SELECT unix_timestamp(EventTime) - SECOND(EventTime), count(*) FROM hits_100m WHERE CounterID = 34 AND EventDate >= TIMESTAMP('2013-07-01') AND EventDate <= TIMESTAMP('2013-07-02') AND NOT Refresh != 0 AND NOT DontCountHits != 0 GROUP BY m ORDER BY m;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_8496@mturlrep13_201309180211_1815785181.txt
hive> SELECT unix_timestamp(EventTime) - SECOND(EventTime), count(*) FROM hits_100m WHERE CounterID = 34 AND EventDate >= TIMESTAMP('2013-07-01') AND EventDate <= TIMESTAMP('2013-07-02') AND NOT Refresh != 0 AND NOT DontCountHits != 0 GROUP BY m ORDER BY m; ;
hive> quit;
times: 3
query: SELECT unix_timestamp(EventTime) - SECOND(EventTime), count(*) FROM hits_100m WHERE CounterID = 34 AND EventDate >= TIMESTAMP('2013-07-01') AND EventDate <= TIMESTAMP('2013-07-02') AND NOT Refresh != 0 AND NOT DontCountHits != 0 GROUP BY m ORDER BY m;
spawn hive
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j.properties
Hive history file=/tmp/kartavyy/hive_job_log_kartavyy_8703@mturlrep13_201309180211_1151158687.txt
hive> SELECT unix_timestamp(EventTime) - SECOND(EventTime), count(*) FROM hits_100m WHERE CounterID = 34 AND EventDate >= TIMESTAMP('2013-07-01') AND EventDate <= TIMESTAMP('2013-07-02') AND NOT Refresh != 0 AND NOT DontCountHits != 0 GROUP BY m ORDER BY m; ;
hive> quit;
stop time: Ср. сент. 18 02:11:52 MSK 2013
Reference in New Issue View Git Blame Copy Permalink