Warrior Release Compatibility

[ahcbb6]

Hello,

The following set of patches, provide compatibility to build with the warrior branch from Yocto,
it is important to note that this upgrades the branch three releases ahead, from rocko-> sumo->thud -> warrior.

More information about what each of these upgrades bring by themselves can be found at:
Sumo Release Notes
Thud Release Notes
Warrior Release Notes

Initially (on the first pull request) this was only able to build the fb-openbmc-image along with DISTRO="openbmc-fb".

It wasn't very clear which BSPs were supported at the beginning but after I saw the first pull request I could figure out the way the Travis CI worked and what was being tested, I was able to imitate the Travis CI flow so now the Warrior branch compatibility has been extended to build on:

BSPs/Layers (and their images) now compatible with warrior:
- wedge
- wedge100
- yosemite
- lightning
- cmm
- galaxy100
- fbtp
- fbttn
- fby2

For obvious reasons this provides many improvements in the system, including many CVEs and fixes from packages such as openssl, busybox, etc.

The system now correctly builds and runs a kernel version 5.0.7 (linux-yocto)

There are a couple of necessary changes related to python3 because of the upgrade of the manifest generation / python3 packaging system itself (which I personally did on oe-core) 8d94b9 and b677787

initscripts required some changes as well since it should be getting the ip command from iproute2 and not from busybox as it was happening before, also rdepend on dig since it is being used on the hostname script.

DISTRO wise, there was some required changes to DISTRO_FEATURES due to changes in OE-Core.
9d973d3

This also contains many fixes to several packages including:
- u-boot-2016.07
- fw-util
- bios-util
- ipmitool
- vboot-utils
- tpm-tools
- trousers
- u-boot-tools
- libkv
- ast-mdio
- openbmc-gpio
- opkg_utils
- dhcp
- bind
- initscripts

I would be open to discuss each of these individually if the commit message on each of them is not clear, but discussing each of these in the cover letter is not feasible.

The only downside of these set of commits is that it contains a temporary fix, because, since the inclusion of security flags on Yocto on: 30f9a507 and the introduction of GCC 8, a lot of more security warnings are shown, and when using -Wall this causes a error while building several packages, I decided to squash all these in a single commit, to force them all not to work if this commit is not included, this way we'll be forced to analyze each of this cases individually, but the correct fix for these individual cases should come from the source code itself.
More info at:
GCC8 - Warnings

There are also a couple of bugs I found while developing these, but I believe it would be better to file them through here, since while I could fix them, I would not be able to make an informed decision on the correct/best way to do so (e.g bios-util LICENSE is most certainly wrong).

This set of commits also upgrades linux-aspeed (1250 and 2520) to work with Linux 5.0.3, along with some dtb/kernel-fitimage fixes

Since I don't have hardware available at the moment, I have tested these at runtime on qemuarmv5 (qemuarm was moved to qemuarmv5 on warrior 6acb45 ) , since it uses the same architecture/tune (arm926ejs) as several machines within the layer such as: mavericks, galaxy, lightning, wedge, wedge100 and yosemite, although it is worth mentioning that qemuarm (v7) does work correctly as well.

AFAIC there could still be some runtime issues, e.g. /mnt/data isnt mounted, like it would when using real hardware (due to the openbmc-utils dependency)

Regards,
Alejandro

[ahcbb6]

Well, I've been monitoring the build and an unrelated error was thrown, I believe the Travis CI infrastructure does not support changes like this, and it tries to protect itself from performing too many jobs.

This is what is shown (tail):
0: python3-3.7.2-r0 do_compile - 318s (pid 26594)
1: gmp-6.1.2-r0 do_compile - 75s (pid 5734)
[Currently 2 running tasks (782 of 3331) 23% |####### |
0: python3-3.7.2-r0 do_compile - 323s (pid 26594)
1: gmp-6.1.2-r0 do_compile - 80s (pid 5734)

The job exceeded the maximum time limit for jobs, and has been terminated.
LOG

This will certainly happen for all other MACHINES/BSPs as well.

I believe that its caused because usually, most of the Shared State is already available and its not necessary to perform this many jobs, in this case though, since you don't build for the warrior branch (and its two releases ahead) you don't have the required shared state already available, its basically trying to build most if not everything from scratch.

Is there any way I can make a special request in which the limit of number of jobs is modified?, I have already tested all the listed MACHINES and images myself, but I'm a bit disappointed by this, because I wanted you to see it working too, if possible, I'd like this to be tested on the Travis CI.

Thanks!

Alejandro

[ahcbb6]

Hello @amithash , @benwei13 , can you modify the Travis CI so the build is able to continue?, like I said I've already built everything by imitating what the Travis CI does myself, but I'd like to see it built on your infrastructure anyway.

Thanks!

Alejandro

[amithash]

@aehs29 This is super cool!!! I agree, we probably should fix the compiler warnings. I am ok getting the temp cflags to disable said warnings and fixing them one by one and removing the exception.

For fw-util, Let me work on deleting the 0.1 version of fw-util so you don't have to copy-paste the entire source! :-( Once I do that you can rebase your tree.

PS: Our auto tooling is probably better at getting PRs one diff at a time.

[ahcbb6]

Awesome!, yeah absolutely, I can definitely rebase my tree on top of that, the only reason I didn't delete the fw-utils_0.1 recipe myself is because I didn't want to make a misinformed decision that would break someone else's flow (perhaps something/someone was still using it but I couldn't be sure), but yeah, it'd be a lot better if we have it in a common place.

I have a similar issue with the kernel_fitimage class (coming from meta-aspeed) vs kernel-fitimage class (coming from oe-core), in most situations I would rather use the one from oe-core, but the fact that it is named differently (looks like on purpose) makes me think there is a reason why we don't use the one from oe-core, I just, at this point don't know why, and decided to keep and fix the current one for now.

[amithash]

I went ahead and deleted 0.1 version of fw-util from meta-yosemite, meta-fby2, meta-fbttn and meta-minilaketb. I am guessing we wont need this patch (and similar patches for the other platforms) anymore.

[ahcbb6]

Great, just for sanity, should we check the priorities set for BBFILE_PRIOIRITY for the different layers?, what is the reasoning behind the BBFILE_PRIORITY on some layers, for example fby2 has 8 and fbttn has 20, is this something we have to review to make sure its doing what we want?.

Thanks!
Alejandro

[ahcbb6]

Hey guys ( @amithash )

I finally had some time this weekend to play around with this again and solve the pending issues, sorry for the delay.

Changes since the last pull request:

- Rebased on top of latest helium

- Removed the following commits due to our previous conversations:
fw-util: Port the fw-util and fw-util-test recipes to meta-fby2
fw-util: Port the fw-util and fw-util-test recipes to meta-fbttn
These two were no longer necessary since the fw_util_0.1 recipe was
deleted on your part, thanks!.

- initscripts: Fix syslog syntax on hostname.
This one was not necesary due to the usage of rsyslog instead.

Resolved conflicts on:
- CFLAGS: Silence security and gcc8 warnings (temporarily) for several packages
Due to the fw-util_0.1 not existing anymore

- libipmi: Change PREFERRED_VERSION to 0.2
libipmi PREFFERED_VERSION does not exist for cnm anymore

- linux-aspeed: Upgrade to kernel version 5.0.3 for 1250
This commit used to provide a defconfig for kernel v5, but a similar defconfig
has now been merged on helium

- openbmc-fb: Set DISTRO_FEATURES correctly
Package management was recently added on helium, which conflicted with the previous change.
This was rebased on top of that change, and also kept DISTRO_FEATURES_LIBC for compatibility
purposes from the conversation we had, but fixed the ipv4 issue that still remained.

- u-boot: Fix compilation error due to incompatibility with dtc
This commit had to change, due to our previous conversation on which this pull
request should work on both rocko AND warrior, the patch for dtc compatibility
is only required on warrior due to the upgrade in dtc version, so, fix this
by conditionally adding the patch depending on the release that is building.

As a side note, I still believe having separate branches for stable releases might
be easier, since as releases progress they fall apart from each other and will
eventually require more and more workarounds like this, but I do understand that
there might be reasons why we might to keep this way as well.

- Changed LAYERCOMPAT to warrior AND rocko for all the previous layers where it was added

meta-facebook: Set warnings instead of fatal errors for missing bbappends

• I added a commit that sets BB_DANGLINGAPPENDS_WARNONLY to change default behavior of
  missing bb files to a respective bbappend, because the recipe u-boot-tools which requires a bbappend did not exist on the rocko release for example.

--------------------
ERROR: No recipes available for:
rocko/poky/../../../meta-aspeed/recipes-bsp/u-boot/u-boot-tools_%.bbappend
--------------------

Default behavior causes the cooker to exit with a fatal error, but setting it to show warnings
allows the build to keep going.
Currently I set this on meta-facebook layer.conf since its pretty global, otherwise it would
have to be set on each layer or local.conf.sample individually.

Keep rocko as DEFAULT_DISTRO

• I deleted the commit that set warrior as DEFAULT_DISTRO, again this change was because I thought there would be a separate branch for warrior itself, so, for now, rocko is kept as default.

Added Testing script

• I put a temporary commit at the top, on which a script is added to naively emulate
  what Travis CI does (since we know this many changes breaks it), it basically builds for
  the same default boards Travis does (more BOARDS could be added if necessary using the
  BOARDS variable), but does it sequentially, downloads are shared amongst builds to speed up the process a little bit, but this obviously still takes a while.

Usage is very simple:

$ BOARDS="wedge" ./simulate_travis_ci_build.sh # builds wedge board on DISTRO_DEFAULT (rocko)
$ DISTRO_DEFAULT="warrior" BOARDS="yosemite fby2" ./simulate_travis_ci_build.sh  # Builds both yosemite and fby2 for warrior release
$ DISTRO_DEFAULT="warrior" ./simulate_travis_ci_build.sh # Builds all boards that are used as default on Travis CI for warrior release

• The Shared State Cache issue that I previously mentioned can be easily seen when building sequentially, as I said debugging the sstate shows this is caused because OPENBMC_VERSION invalidates tasks in openssh and that causes a chain reaction on all dependent tasks.

- I've tested builds for all the Travis CI default boards both on rocko and warrior, from scratch and using the sstate cache as well.

[ahcbb6]

Great I think the Travis CI passed (for the most part haha), Its getting an unrelated error when installing packages on the Ubuntu host:

Build 266
Failed on lightining:

apt-get install failed
The command "sudo -E apt-get -yq --no-install-suggests --no-install-recommends $(travis_apt_get_options) install build-essential wget diffstat chrpath texinfo bzip2 libthread-queue-perl" failed and exited with 100 during .

Even build 265 passed, but failed on wedge with a similar error:
apt-get install failed
The command "sudo -E apt-get -yq --no-install-suggests --no-install-recommends $(travis_apt_get_options) install build-essential wget diffstat chrpath texinfo bzip2 libthread-queue-perl" failed and exited with 100 during .

Juts pointing out that this did not break the Travis CI as before, since the DEFAULT_DISTRO is still set to rocko, so you would still need to use the provided script to test for warrior manually, but it does show that my warrior compatibility changes don't break rocko builds which is great!

Check it when you get a chance and let me know @amithash !

Thanks!
Alejandro

[ahcbb6]

I added additional support for the following boards:
- wedge400
- fby3
- yamp
- minilaketb
- minipack

Even though these boards are not tested automatically by Travis

PD: Also, it looks like Travis is still having network issues