powered by simpleCommunicator - 2.0.49     © 2025 Programmizd 02
Форумы / NoSQL, Big Data [игнор отключен] [закрыт для гостей] / Запуск job'a в gobblin'e
1 сообщений из 1, страница 1 из 1
Запуск job'a в gobblin'e
    #39783150
x17.mstu
Скрыть профиль Поместить в игнор-лист Сообщения автора в теме
Участник
Запускаю job в gobblin'e
Код: plsql
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.
26.
27.
28.
29.
30.
31.
32.
33.
34.
35.
36.
37.
38.
39.
40.
41.
42.
43.
44.
45.
46.
47.
48.
49.
50.
51.
52.
53.
54.
55.
56.
57.
58.
59.
60.
61.
62.
63.
64.
65.
66.
67.
68.
69.
70.
71.
72.
73.
74.
75.
76.
77.
78.
79.
80.
81.
Результат журналов логов vim gobblin-current.log
2019-03-06 14:01:11 UTC INFO  [SchedulerService STARTING] org.quartz.impl.StdSchedulerFactory  1327 - Quartz scheduler 'LocalJobScheduler' initialized from specified file: '/opt/gobblin/gobblin-dist/conf/quartz.properties'
2019-03-06 14:01:11 UTC INFO  [SchedulerService STARTING] org.quartz.impl.StdSchedulerFactory  1331 - Quartz scheduler version: 2.2.3
2019-03-06 14:01:11 UTC INFO  [SchedulerService STARTING] org.quartz.core.QuartzScheduler  575 - Scheduler LocalJobScheduler_$_NON_CLUSTERED started.
2019-03-06 14:01:12 UTC WARN  [JobScheduler STARTING] org.apache.hadoop.util.NativeCodeLoader  62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2019-03-06 14:01:13 UTC INFO  [JobScheduler STARTING] org.apache.gobblin.scheduler.JobScheduler  507 - Scheduling configured jobs
2019-03-06 14:01:14 UTC INFO  [JobScheduler STARTING] org.apache.gobblin.scheduler.JobScheduler  527 - Loaded 1 job configurations
2019-03-06 14:01:14 UTC INFO  [JobScheduler-0] org.apache.gobblin.util.ClustersNames  74 - no default cluster mapping found
2019-03-06 14:01:14 UTC WARN  [JobScheduler-0] org.apache.gobblin.runtime.AbstractJobLauncher  223 - Creating a job specific SharedResourcesBroker. Objects will only be shared at the job level.
2019-03-06 14:01:17 UTC INFO  [JobScheduler-0] org.reflections.Reflections  229 - Reflections took 2597 ms to scan 48 urls, producing 809 keys and 2849 values
2019-03-06 14:01:18 UTC ERROR [JobScheduler-0] org.apache.gobblin.scheduler.JobScheduler$NonScheduledJobRunner  629 - Failed to run job GobblinKafkaQuickStart
org.apache.gobblin.runtime.JobException: Failed to run job GobblinKafkaQuickStart
        at org.apache.gobblin.scheduler.JobScheduler.runJob(JobScheduler.java:437)
        at org.apache.gobblin.scheduler.JobScheduler$NonScheduledJobRunner.run(JobScheduler.java:627)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: Failed to create job launcher: java.lang.RuntimeException: Failed to create FsDatasetStateStore with factory
        at org.apache.gobblin.runtime.JobLauncherFactory.newJobLauncher(JobLauncherFactory.java:158)
        at org.apache.gobblin.runtime.JobLauncherFactory.newJobLauncher(JobLauncherFactory.java:107)
        at org.apache.gobblin.runtime.JobLauncherFactory.newJobLauncher(JobLauncherFactory.java:85)
        at org.apache.gobblin.runtime.JobLauncherFactory.newJobLauncher(JobLauncherFactory.java:68)
        at org.apache.gobblin.scheduler.JobScheduler.buildJobLauncher(JobScheduler.java:442)
        at org.apache.gobblin.scheduler.JobScheduler.runJob(JobScheduler.java:435)
        ... 4 more
Caused by: java.lang.RuntimeException: Failed to create FsDatasetStateStore with factory
        at org.apache.gobblin.runtime.FsDatasetStateStoreFactory.createStateStore(FsDatasetStateStoreFactory.java:31)
        at org.apache.gobblin.runtime.JobContext.createStateStore(JobContext.java:200)
        at org.apache.gobblin.runtime.JobContext.<init>(JobContext.java:147)
        at org.apache.gobblin.runtime.AbstractJobLauncher.<init>(AbstractJobLauncher.java:183)
        at org.apache.gobblin.runtime.local.LocalJobLauncher.<init>(LocalJobLauncher.java:81)
        at org.apache.gobblin.runtime.JobLauncherFactory.newJobLauncher(JobLauncherFactory.java:144)
        ... 9 more
Caused by: java.lang.RuntimeException: java.io.IOException: No FileSystem for scheme: instance-1.europe-west1-b.c.absolute-axis-232916.internal
        at org.apache.gobblin.runtime.FsDatasetStateStore.createStateStore(FsDatasetStateStore.java:133)
        at org.apache.gobblin.runtime.FsDatasetStateStoreFactory.createStateStore(FsDatasetStateStoreFactory.java:29)
        ... 14 more
Caused by: java.io.IOException: No FileSystem for scheme: instance-1.europe-west1-b.c.absolute-axis-232916.internal
        at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2304)
        at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2311)
        at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:90)
        at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2350)
        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2332)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:369)
        at org.apache.gobblin.runtime.FsDatasetStateStore.createStateStore(FsDatasetStateStore.java:107)
        ... 15 more

Файл настройки джоба.
kafka-gobblin-hdfs.pull

###################### job configuration file ######################
job.name=GobblinKafkaQuickStart
job.group=GobblinKafka
job.description=Gobblin quick start job for Kafka
job.lock.enabled=false

kafka.brokers=0.0.0.0:6667
source.class=gobblin.source.extractor.extract.kafka.KafkaSimpleSource
extract.namespace=gobblin.extract.kafka

writer.builder.class=gobblin.writer.SimpleDataWriterBuilder
writer.file.path.type=tablename
writer.destination.type=HDFS
writer.output.format=txt

data.publisher.type=gobblin.publisher.BaseDataPublisher

mr.job.max.mappers=1
metrics.reporting.file.enabled=true
metrics.log.dir=/gobblin-kafka/metrics
metrics.reporting.file.suffix=txt

bootstrap.with.offset=earliest
fs.uri=instance-1.europe-west1-b.c.absolute-axis-232916.internal:8020;
writer.fs.uri=instance-1.europe-west1-b.c.absolute-axis-232916.internal:8020;
state.store.fs.uri=instance-1.europe-west1-b.c.absolute-axis-232916.internal:8020;

mr.job.root.dir=/opt/gobblin/gobblin-dist/job_work/working
state.store.dir=/opt/gobblin/gobblin-dist/job_work/state-store
task.data.root.dir=/opt/gobblin/gobblin-dist/job_work/task-data
data.publisher.final.dir=/opt/gobblin/output


Помогите понять в чем может быть ошибка ?
...
Рейтинг: 0 / 0
1 сообщений из 1, страница 1 из 1
Форумы / NoSQL, Big Data [игнор отключен] [закрыт для гостей] / Запуск job'a в gobblin'e
Целевая тема:
Создать новую тему:
Автор:
Закрыть
Цитировать
Найденые пользователи ...
Разблокировать пользователей ...
Читали форум (0):
Пользователи онлайн (0):
x
x
Закрыть


Просмотр
0 / 0
Close
Debug Console [Select Text]