Invalid Statistics type for Load job

Hi,
I have following code snippet in my project.

Credentials c = new ServiceAccountCredentials();  /* hiding credentials here*/
 BigQueryOptions options = BigQueryOptions.newBuilder().setCredentials(c).setProjectId('id').build();
 BigQuery bq = options.getService();
 TableId id = TableId.of('project id', 'dataset id', 'table id');
 Table t = bq.getTable(id);
 t.load(FormatOptions.csv(), "gs://bucket/file1.csv");
 System.out.println(loadJob.getStatistics());

I was expecting the statistics to be of 'Load' type but its of 'Copy' type.

Can anyone tell me where am I doing wrong? Indeed, data is getting inserted into table but I am missing the output statistics (no of rows, no of bytes, etc) due to wrong type being returned.

@Syed-SnapLogic

Hi @Syed-SnapLogic, I don't think you are doing anything wrong. I tried to reproduce your issue and I have exactly same result: insert a load job, but receive CopyStatistics.

What happens here is the following: on the moment when the job statistics object should be created, there is statistics type info missing (either it is not returned from the server or is lost somewhere in between), and when there is no statistics type present, the google-cloud-java bigquery client defaults to CopyStatistics (check JobStatistics.fromPb() method for details). When that code is executed the actual statisticsPb object has only 2 fields: creationTime, startTime (i.e. no type info at all).

I need to investigate more why that happens, but there is a good chance that it is either bug on the client or the server side.

@vam-google

Hi @vam-google, I found the resolution. You can still keep your hunt if you wish.
If I say,

Job loadJob = table.load(...);
System.out.println(loadJob.getStatistics());

then certainly it would give copy statistics. However, doing

Job loadJob = table.load(...);
 loadJob = loadJob.waitFor();
System.out.println(loadJob.getStatistics());

is giving load statistics.

Thanks for the update! It is good that it works for you now, but in that case you are dealing with an already completed job. In other words if the job is completed, it returns the proper statistics type, if it is in progress it returns a wrong one. This is still not good.

bigquery: properly detect statistics type (#2742)

chore: Update ci.yaml to setup Java 21 unit tests (#2357)

deps: update dependency com.google.api.grpc:proto-google-cloud-trace-…

chore(main): release 2.35.0 (googleapis#2335)

build(deps): update dependency org.apache.maven.plugins:maven-project…

chore(main): release 3.40.1-SNAPSHOT (#2357)

Navigation Menu

Repository navigation

Invalid Statistics type for Load job #2357

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Invalid Statistics type for Load job #2357

Description

Activity

vam-google commented on Aug 28, 2017

Syed-SnapLogic commented on Aug 30, 2017

vam-google commented on Aug 30, 2017

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions