V8 CI is not working

@mhdawson

Looks like an error with glient fetching v8

Fetching V8 from chromium.googlesource.com
Error: Command 'download_from_google_storage --no_resume --platform=linux* --no_auth --bucket chromium-clang-format -s v8/buildtools/linux64/clang-format.sha1' returned non-zero exit status 1 in /home/iojs/build/workspace/node-test-commit-v8-linux/nodes/ppcle-ubuntu1404/v8test/v8test/deps
Running: gclient root
Running: gclient config --spec 'solutions = [
  {
    "url": "https://chromium.googlesource.com/v8/v8.git",
    "managed": False,
    "name": "v8",
    "deps_file": "DEPS",
    "custom_deps": {},
  },
]
'
Running: gclient sync --with_branch_heads
Traceback (most recent call last):
  File "/home/iojs/build/workspace/node-test-commit-v8-linux/nodes/ppcle-ubuntu1404/v8test/v8test/depot_tools/fetch.py", line 299, in <module>
    sys.exit(main())
  File "/home/iojs/build/workspace/node-test-commit-v8-linux/nodes/ppcle-ubuntu1404/v8test/v8test/depot_tools/fetch.py", line 294, in main
    return run(options, spec, root)
  File "/home/iojs/build/workspace/node-test-commit-v8-linux/nodes/ppcle-ubuntu1404/v8test/v8test/depot_tools/fetch.py", line 288, in run
    return checkout.init()
  File "/home/iojs/build/workspace/node-test-commit-v8-linux/nodes/ppcle-ubuntu1404/v8test/v8test/depot_tools/fetch.py", line 131, in init
    self.run_gclient(*sync_cmd)
  File "/home/iojs/build/workspace/node-test-commit-v8-linux/nodes/ppcle-ubuntu1404/v8test/v8test/depot_tools/fetch.py", line 78, in run_gclient
    return self.run(cmd_prefix + cmd, **kwargs)
  File "/home/iojs/build/workspace/node-test-commit-v8-linux/nodes/ppcle-ubuntu1404/v8test/v8test/depot_tools/fetch.py", line 68, in run
    return subprocess.check_output(cmd, **kwargs)
  File "/usr/lib/python2.7/subprocess.py", line 573, in check_output
    raise CalledProcessError(retcode, cmd, output=output)
subprocess.CalledProcessError: Command '('gclient', 'sync', '--with_branch_heads')' returned non-zero exit status 2
V8 fetch failed

/cc @mhdawson @nodejs/v8

Worth mentioning this is happening for any + all attempts to run CI right now

@joransiu

To be clear, this is happening for all runs of node-test-commit-v8-linux, not node-test-commit right?

@joransiu @jBarz do I recall you guys talking about something like this on Slack?

@john-yan

This started failing after https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/656565.
@john-yan is working with chromium's infra team on this https://bugs.chromium.org/p/chromium/issues/detail?id=764087 . In the meantime could add the following to the jenkins job

cd <path>/depot_tools
git revert -n 509776e
cd -
export DEPOT_TOOLS_UPDATE=0

Have put the work around into the job and kicked off a run to see if it resolves the issue.

Assuming it's https://ci.nodejs.org/job/node-test-commit-v8-linux/912/ then it looks like it worked.

We should leave this open to track the proper fix though, sounds like John is making progress.

Looks like work around did the trick, lets leave this open until the proper fix is in place.

@jBarz

@jBarz @joransiu this looks different than this issue https://bugs.chromium.org/p/chromium/issues/detail?id=764087&can=2&start=0&num=100&q=&colspec=ID%20Pri%20M%20Stars%20ReleaseBlock%20Component%20Status%20Owner%20Summary%20OS%20Modified&groupby=&sort=

But it did remind me to ask where we are on this one. At some point I should remove the work around.

It should be fixed now. All we need to do is adding an extra env variable.
export VPYTHON_BYPASS="manually managed python not supported by chrome operations"

Looks like the temporary patch is now failing 😭

I'm hacking on the CI job right now to make a permanent fix

Was not able to fix it. I attempted to revert all of the hacks we had currently put in and added export VPYTHON_BYPASS="manually managed python not supported by chrome operations"

Same error reported in OP

~~The error I'm seeing is:~~ env issue

Using https://ci.nodejs.org/job/node-test-commit-v8-linux-fedora24/nodes=fedora24,v8test=v8test/398/console as reference.
Manual on test-rackspace-fedora24-x64-1

Ok found some stuff:

The fedora machines (https://ci.nodejs.org/job/node-test-commit-v8-linux-fedora24/) are failing because they don't have bzip2 (and their yum repos have been archived to http://archives.fedoraproject.org/pub/archive/fedora/linux), but I see this is only run by timer...

ppc getting:

iojs@test-osuosl-ubuntu14-ppc64-be-3:~/build/workspace/node-test-commit-v8-linux/nodes/ppcbe-
ubuntu1404/v8test/v8test/depot_tools$ ./update_depot_tools
UNKNOWN Machine architecture: ppc64

which lead me to https://bugs.chromium.org/p/chromium/issues/detail?id=764087.
I think it's a depot_tools regression of that issue.

The best I got is to revive the test on the fedora24 - https://ci.nodejs.org/job/node-test-commit-v8-linux-fedora24/399/nodes=fedora24,v8test=v8test/

@refack

@refack were you testing with VPYTHON_BYPASS="manually managed python not supported by chrome operations"

@refack

@refack were you testing with VPYTHON_BYPASS="manually managed python not supported by chrome operations"

In multiple variations with no success (with " without, in the same line, with export, before make after make, with just calling gclient sync --with_branch_heads).
Also it wasn't there yesterday when the last green job run https://ci.nodejs.org/view/All/job/node-test-commit-v8-linux/953/nodes=ppcbe-ubuntu1404,v8test=v8test/consoleFull

But now I see that we used to roll back depot_tools, and it broke sometime in the last 24h:

+ git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git
Cloning into 'depot_tools'...
+ cd depot_tools
+ git revert -n 509776e
error: could not revert 509776e... [gsutil] run through "vpython" (2)
hint: after resolving the conflicts, mark the corrected paths

Holly molly! It's green https://ci.nodejs.org/view/All/job/node-test-commit-v8-linux/968/

Looks like the git changes broke stuff... failures across all environments rn

@MylesBorins

@MylesBorins that's a v4.x PR right? (https://ci.nodejs.org/view/All/job/node-test-commit-v8-linux/970/)
Do we know when was the last known good revision for testing v4.x?

nvm... I think I broke stuff (in my branch)... Will check in again in am and confirm

@john-yan

@john-yan I tried putting in the final fix by adding

# temporary work around for https://github.com/nodejs/build/issues/886
#export DEPOT_TOOLS_UPDATE=0 
export VPYTHON_BYPASS="manually managed python not supported by chrome operations"

But it is still failing on the PPC platforms. See this job which is a clone of the regular CI job with the change in place:

https://ci.nodejs.org/view/All/job/node-test-commit-v8-linux-test-mdawson/configure

Is it just me or does it look like the job is no longer running all of the tests ?

Ignore my last comment, its just that they run in the middle of the job since there is recompilation for the hashseed test.

open https://bugs.chromium.org/p/chromium/issues/detail?id=773857 to fix depot_tools on ppc

Hello, depot_tools should be fixed on ppc linux now.

>fetch v8
Running: gclient root
Running: gclient config --spec 'solutions = [
  { 
    "url": "https://chromium.googlesource.com/v8/v8.git",
    "managed": False,
    "name": "v8",
    "deps_file": "DEPS",
    "custom_deps": {},
  },
]
'
Running: gclient sync --with_branch_heads
Running: git submodule foreach 'git config -f $toplevel/.git/config submodule.$name.ignore all'
Running: git config --add remote.origin.fetch '+refs/tags/*:refs/tags/*'
Running: git config diff.ignoreSubmodules all

@mhdawson

crbug pretty much resolved. we can remove the git checkout and git revert and instead add:
export VPYTHON_BYPASS="manually managed python not supported by chrome operations"

/cc @mhdawson @gibfahn

@john-yan

Thank @john-yan 🎉

Tested here: https://ci.nodejs.org/view/All/job/node-test-commit-v8-linux-test-mdawson/8/

Main job updated, run to validate works as expected: https://ci.nodejs.org/job/node-test-commit-v8-linux/999/

Looks good closing.

Navigation Menu

Repository navigation

V8 CI is not working #886

11 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

V8 CI is not working #886

Description

Activity

MylesBorins commented on Sep 16, 2017

gibfahn commented on Sep 16, 2017

jbajwa commented on Sep 17, 2017

mhdawson commented on Sep 18, 2017

gibfahn commented on Sep 18, 2017

mhdawson commented on Sep 18, 2017

mhdawson commented on Oct 3, 2017

john-yan commented on Oct 3, 2017

MylesBorins commented on Oct 9, 2017

MylesBorins commented on Oct 9, 2017

refack commented on Oct 9, 2017

refack commented on Oct 9, 2017

MylesBorins commented on Oct 9, 2017

refack commented on Oct 9, 2017

11 remaining items

refack commented on Oct 10, 2017

MylesBorins commented on Oct 10, 2017

refack commented on Oct 10, 2017

MylesBorins commented on Oct 11, 2017

mhdawson commented on Oct 11, 2017

mhdawson commented on Oct 11, 2017

mhdawson commented on Oct 11, 2017

john-yan commented on Oct 11, 2017

john-yan commented on Oct 17, 2017

refack commented on Oct 17, 2017

refack commented on Oct 17, 2017

mhdawson commented on Oct 18, 2017

mhdawson commented on Oct 18, 2017

mhdawson commented on Oct 18, 2017

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions