14:00:56 <Werner> #startmeeting Release meeting 22.08
14:00:56 <ArmbianHelper> Meeting started Sat Aug 13 14:00:56 2022 UTC.  The chair is Werner. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:00:56 <ArmbianHelper> Useful Commands: #action #agreed #help #info #idea #link #topic.
14:01:08 <IgorPec> #topic check-in
14:01:14 <Heisath> Here.
14:01:22 <Werner> #topic check-in
14:01:23 <teknoid_> hello
14:01:47 <Armbian-Discord> <r​pardini> Hello!
14:02:17 <Armbian-Discord> <08T​onymac32> hello
14:02:21 <[TheBug]> Hey rpardini!
14:02:26 <[TheBug]> Hey teknoid_!
14:02:31 <rpardini> Also hello here
14:02:51 <Armbian-Discord> <a​deepv> here)
14:02:53 <Armbian-Discord> <M​anoftheSea> Does it read people in discord?
14:03:03 <rpardini> it does
14:03:04 <[TheBug]> yep there is bridge
14:03:30 <Armbian-Discord> <M​anoftheSea> Change from May?
14:03:42 <[TheBug]> Wait a few more minutes for straglers then @IgorPec can kick things off :)
14:04:20 <IgorPec> changes are always huge or a lot bigger than we can handle them
14:04:28 <IgorPec> but that is not the only problem we have
14:04:39 <Armbian-Discord> <08T​onymac32> yep, we're changing to all vendor kernels
14:04:42 <Armbian-Discord> <08T​onymac32> the older the better
14:04:47 <Armbian-Discord> <M​anoftheSea> I mean the bridge didn't read me in May
14:05:12 <[TheBug]> BTW Thanks to everyone who has taken time out of their weekend to be here and help support Armbian.  I just want you to know that we appriciate you all and you time and dedication to Armbian!
14:05:21 <IgorPec> we have to think over the whole release model principle
14:06:02 <IgorPec> if we want to proceede with the same level (well tested release, well prepared release) ... the
14:06:11 <IgorPec> then this means serious work
14:06:23 <rpardini> I've just rebased 3-4 weeks of master onto -next. It's mostly moving edge's to 5.19, some new vendor-kernel-based boards
14:06:49 <IgorPec> yes, we have a lot of happening, but just a few days in total to do something.
14:06:55 <rpardini> lotsa rootfs cache changes, otherwise more of the same stuff. some board-side scripts might need a lot of testing
14:07:33 <rpardini> but I've a feeling we could focus testing on 1 or 2 boards per-family and have a good general indication
14:07:33 <IgorPec> well, from developers perspective we have "good stability"
14:07:42 <Werner> #topic release preparation and discussion
14:07:56 <teknoid_> my two boards I support are in productive use - for me its hard to do full image testing 4 times per year as they are built into enclosures
14:08:10 <IgorPec> board count is under better control since last year, but board count solo doesn't help much
14:08:16 <teknoid_> testing kernel is no problem, but full image is a big problem
14:08:26 <IgorPec> for testing kernels we do have automation
14:08:31 <Heisath> teknoid get more boards then. Testing in production is shit.
14:08:33 <Armbian-Discord> <08T​onymac32> uboot often brings challenges and must be tested fresh
14:08:36 <IgorPec> but is not well maintained
14:08:47 <IgorPec> yep, u-boot
14:08:58 <IgorPec> idea is to move many many boards to most recent u-boot
14:09:02 <teknoid_> these boards are out of production for years  ;-)
14:09:10 <rpardini> I haven't seen meson64 u-boot changes. I guess ya'll mean rockchip's u-boot?
14:09:25 <IgorPec> just beeing sure that this goes as expected is complex job
14:09:28 <rpardini> (mostly... of course)
14:09:32 <Armbian-Discord> <08T​onymac32> just in general.  Testing a new image by just the kernel misses everything
14:09:44 <IgorPec> meson is pretty data i think
14:09:49 <rpardini> oh indeed
14:09:51 <Armbian-Discord> <08T​onymac32> I've been guilty of that and got burned
14:09:56 <IgorPec> but we have allwinner and rockchip which is old
14:10:11 <IgorPec> also it would be sane to move to EFI in order to use some standard tools
14:10:22 <Armbian-Discord> <a​deepv> rpardini i think we can just move meson64 uboot to 2022.07
14:10:22 <Armbian-Discord> <08T​onymac32> allwinner needs bumped, I can voich for GX/GXL, but the newer stuff is...
14:10:31 <Armbian-Discord> <08T​onymac32> vouch*
14:10:36 <rpardini> adeepv: indeed. I've that feeling as well
14:10:42 <Armbian-Discord> <08T​onymac32> amlgoic.  #morecoffee
14:10:43 <IgorPec> yes, just the question is when and how to test?
14:11:24 <IgorPec> project "Regular release" has to be well prepared and lead. Neither me or TheBug can pick this up in "by the way"
14:11:54 <rpardini> yeah. I suppose when developers _also_ have to test we get bogged down very quickly
14:11:55 <IgorPec> are 4 full stream releases too much perhaps?
14:12:15 <IgorPec> testings is getting organised better, we do have more voolonteers for this job
14:12:20 <teknoid_> my proposal is to align releases with debian / ubuntu release cycle
14:12:23 <Armbian-Discord> <M​anoftheSea> Moving to EFI means u-boot supporting EFI, right? Seems beyond armbian control
14:12:34 <Armbian-Discord> <08T​onymac32> u-boot does support EFI
14:12:38 <IgorPec> but again, someone needs to lead them. Currently Bug is trying to cover that to some degree
14:12:56 <[TheBug]> Yes, we have been looking for someone to take on this task, in fact we are discussing trying to come up with enough money for a stipend to give to the 'Release Manager' and maybe even try to keep a single manager for more than one release if they are good with the job.  If this is something any of you would have an interest in specifically, please reach out to me directly after the meeting!
14:13:09 <Werner> Regarding the 4 release question from Igor: IMHO yes. Kernel is kind a rolling anyways and two month until next freeze...barely anything can be done within that time window
14:13:29 <Armbian-Discord> <a​deepv> IgorPec Maybe two releases per year would be better?
14:13:37 <Heisath> Agree. I second 2 releases.
14:13:44 <IgorPec> we heard many ideas, like 2 big, 2 small
14:13:56 <IgorPec> small = like no kernel bump
14:14:06 <rpardini> I think we need one release that makes things really stable first... then we can move to 2/year
14:14:07 <IgorPec> staying on the same version, just patch bump
14:14:24 <[TheBug]> Yes I am not sure we really came up with a schedule for 'minor releases' or what that actually means, but maybe if we can define this we could make more 'minor' releases and less 'major' releases?
14:14:39 <IgorPec> really stable can only be achived with massively improved testings . to at least know what is wrong
14:14:55 <rpardini> in the end the release cadence should be dictated by our testing frequency / trustworthiness
14:15:02 <IgorPec> but currently testings is - down to developers mainly
14:15:44 <[TheBug]> BTW https://www.armbian.com/rc-testing/ is our first attempt at being able to track testing -- if you guys find that there could be improvements here, please communicate them to me so I can see if I can continue to improve this form / process
14:15:45 <ArmbianHelper> ^ RC Testing – Armbian
14:16:35 <[TheBug]> This is also important because it will submit all of these testing into a single Jira ticeket which all can view including release manager: https://armbian.atlassian.net/browse/AR-1271
14:16:36 <ArmbianHelper> 4AR-1271 6[Task] "RC Testing for August 2022 Release" reported by 3TheLinuxBug at 2022-07-27. Status: To Do
14:16:37 <ArmbianHelper> ^ [AR-1271] - Jira
14:16:42 <IgorPec> also there are ideas to have two repositories
14:16:44 <rpardini> very interesting TheBug. Maybe worth it to do a on-board script that reports stuff like armbianmonitor and the answers to test questions
14:16:54 <IgorPec> one rolling community, basically current trunk
14:17:19 <IgorPec> move everything there and have pro grade branch with a lot more testings
14:17:25 <rpardini> IgorPec: makes sense. since we have 0 testing capacity, nightly-only always-dirty releases are all we can afford
14:17:28 <[TheBug]> (please note the ticket number will change every release in case you were planning to bookmark that)
14:17:59 <[TheBug]> rpardini: if you would be interested in helping design that and come up with an easy method for submission of the information I am fully on board with that idea
14:18:28 <IgorPec> basic testing capacity is this https://github.com/armbian/scripts/actions/runs/2544689819
14:18:29 <ArmbianHelper> ^ Smoke tests · armbian/scripts@8942200 · GitHub
14:19:10 <rpardini> TheBug: yes. Count me in. (goes together with my build-log uploader, firstrun-rewrite, and other initiatives)
14:19:35 <[TheBug]> rpardini: excelent.  We will discuss this more in the near future then.
14:19:43 <IgorPec> but before developing this further, I think "release manager" is quite a serious role here
14:19:47 <[TheBug]> indeed.
14:20:04 <rpardini> yeah, I've never been release manager, and have no idea what it really entails
14:20:17 <IgorPec> it means followint the agenda and talking to people
14:20:40 <IgorPec> we do have agenda, perhaps its not the best, but it gets us where we want
14:20:40 <[TheBug]> Also if you guys don't feel like any of you would have time, but know someone who could fit the role looking for some part-time work that would take it serious, please send them our way.  It doesn't have to be one of y ou take on this but we do need someone who has enough skillsets to work with everyone and help to make the release happen.
14:21:03 <rpardini> what material do we have? agenda?
14:21:08 <IgorPec> https://docs.armbian.com/Process_Release-Model/
14:21:09 <ArmbianHelper> ^ Model - Armbian Documentation
14:21:29 <IgorPec> https://docs.armbian.com/Process_Release-Model/#release-coordinating
14:21:30 <ArmbianHelper> ^ Model - Armbian Documentation
14:21:52 <IgorPec> it might need changes and improvements
14:22:19 <rpardini> Ok but that is already a very good start
14:22:31 <[TheBug]> And if someone wants to take on the role I will work along with  them for the first release and we can work to improve documentation/procedures as that goes.  SO you don't have to be concerned about being all alone
14:22:40 <rpardini> My question was more on the technical side. For example, the RC branch should have `tag:xx` instead of `branch:xxx` right?
14:22:49 <IgorPec> yes, "just" we do not have people that would cover that. but we have a lot of people that would want "stability"
14:23:35 <IgorPec> currently we branched whole build system and build from there
14:24:01 <rpardini> also, I've this in the back of my head, we need some heads-up/buy-in/agreement about the release from `media`, `sunxi` ppl (mostly in Russia I guess). How to communicate with them?
14:24:12 <IgorPec> and those branches stayed there. Since sources were frozen before, in theory, its possible to remake the same image later on.
14:25:06 <rpardini> being able to "freeze" what fetch_from_repo pulls, based on what is the RC/release branch, is a pending feature on my (build system) side
14:25:13 <IgorPec> "well, maintainers has to participate in the release process or board goes out"
14:25:34 <[TheBug]> rpardini: there was some conversation on this last night in fact between Tony, Igor and I, I know also jock has some of these concerns.  I would tell you guys this should be first item in governance, but I think we may need to work on this before that can be implemented.  As such we may want to setup a quick call together to discuss how that can done and invite Oleg as well so we can find
14:25:34 <[TheBug]> some comprimise.
14:25:39 <IgorPec> this is a bit light as we also can't provide what we would like
14:26:41 <IgorPec> but for release process, this is not that critical. Release is "hey folks, lets stabilize this huge system best way possible"
14:26:53 <IgorPec> this is common work per se, but has to be coordinated
14:27:06 <Armbian-Discord> <c​lee> works for me
14:27:11 <rpardini> yeah. again, in the end, most of the pain points can/should be addressed by features in build system. except, of course, getting actual reproducible tests and results done by people who know what they're doing.
14:27:12 <IgorPec> to get maximum outcome
14:28:02 <IgorPec> build system adds its own problems here, but for releasing a distribution, its "just" a tool. which will break down too :)
14:28:24 <rpardini> yep. repo management for example is something 100% out of my reach
14:29:02 <IgorPec> well, this is a complex system and its difficult to get things under any control
14:29:03 <rpardini> also, testing a regular user doing `apt update & apt dist-upgrade` is a much more complicated beast than testing `burn image to SD and it works`
14:29:27 <IgorPec> apt update and upgrade is automatised, but on a clean system
14:29:54 <rpardini> that could also be something to me moved into `community` vs `pro` releases. community would not assure apt-distupgrade upgradeability
14:30:03 <rpardini> while pro might
14:30:27 <Armbian-Discord> <08T​onymac32> $$
14:30:30 <IgorPec> in general. levels of testing should greatly be improved. i am trying and trying but not much interest from general public to hold up into CI automation
14:31:19 <IgorPec> yes, certain things will be moved under pro versions which are interested that thins are in better shape
14:31:30 <rpardini> yeah. I need a decent SD-mux like thingy to work that
14:31:48 <rpardini> other solutions won't reproduce user's experience like switching SD cards would
14:31:51 <IgorPec> or if we would have ability to find sponsors for releases?
14:32:25 <IgorPec> time lost is simply too big if you want to do it right
14:32:36 <rpardini> (eg: I've u-boot based A/B switching working for most bootscript-based boards... but that won't test uboot itself, only from kernel down)
14:32:55 <[TheBug]> rpardini: I think one thing we need to do somewhere is start to document tools which would improve the development process for our developers  and maintainers somewhere and as Armbian can fund it possibly try and help provide those things?
14:33:19 <IgorPec> we can setup a proper lava test system, but we are talking about a serious operation
14:34:23 <IgorPec> so to recap. what we can do with what we have?
14:35:34 <Armbian-Discord> <a​deepv> rpardini I am also thinking about how to automate testing for uboot, but I have no good ideas yet
14:35:36 <IgorPec> what is easy is = build images, test them for corruption, run apt upgrade on 50+ boards
14:35:59 <IgorPec> but this does not label release as "stable" nor "tested"
14:36:53 <Armbian-Discord> <a​deepv> @IgorPec There is another thing: there are changes between releases that are not available by simply updating packages
14:37:38 <Armbian-Discord> <07I​gorPec> yes, i understand that. (automated) images testing would be perfect.
14:37:58 <Armbian-Discord> <07I​gorPec> now, we have more voolonteers that can help testing, but that represent us even more burden
14:38:09 <Armbian-Discord> <07I​gorPec> as someone has to talk with them, encorage them
14:38:39 <Armbian-Discord> <07I​gorPec> and we need to thank them
14:39:03 <Armbian-Discord> <r​pardini> yep. I can propose to do myself 1 or at most 2 rounds of testing on N2, HC4, VIM3L
14:39:20 <Armbian-Discord> <r​pardini> but, only from new-image-to-SD-and-see-if-it-works-for-me style
14:39:56 <Armbian-Discord> <r​pardini> and that's only because I really wanna release 5.19 edge on meson64 😉
14:40:23 <[TheBug]> rpardini: as you have Vim3L maybe you can ping NicoD at some point this coming week and just chat with him on it, I know he is also maintaining and has been trying to help overcome some issues -- Tonymac32 and Tenkawa helped him a bit on previous weeks voice chat, but maybe you could check with him if there are any fixes you could help him get put in place for that board.
14:40:25 <Armbian-Discord> <08T​onymac32> is 5.19 an LTS or not?  It does bring in a lot of good stuff
14:40:35 <Armbian-Discord> <07I​gorPec> 5.19.y as CURRENT?
14:40:47 <Armbian-Discord> <r​pardini> great question, dunno about LTS status of 5.19, or if 5.20 is coming or not
14:40:53 <Armbian-Discord> <a​deepv> 5.15 is LTS
14:40:58 <Armbian-Discord> <07I​gorPec> probably 6.y is next
14:41:01 <Armbian-Discord> <r​pardini> I guess 5.18 is too early for current
14:41:15 <Armbian-Discord> <08T​onymac32> I was part of the "Current should always be LTS" crowd
14:41:20 <rpardini> sorry, meant 5.19
14:41:21 <Armbian-Discord> <07I​gorPec> hmm, but with meson64 we have 5.10 for CURRENT
14:41:28 <Armbian-Discord> <07I​gorPec> which is not very goo
14:41:38 <Armbian-Discord> <r​pardini> yeah, but 5.19 edge is super-new
14:41:44 <Armbian-Discord> <08T​onymac32> yeah that got missed, I think 5.15 was causing troubles with the newer boards
14:41:50 <Armbian-Discord> <M​anoftheSea> 5.20 is 6.0
14:41:57 <Armbian-Discord> <r​pardini> 5.15 meson64 is completely borked
14:42:05 <Armbian-Discord> <r​pardini> 5.11 - 5.18 meson64 is completely borked btw
14:42:18 <Armbian-Discord> <07I​gorPec> than its better to move meson64-CURRENT directly to 5.19.y ?
14:42:22 <Armbian-Discord> <07I​gorPec> right now
14:42:25 <teknoid_> I'll give 5.19 a try on my C2 next week
14:42:38 <Armbian-Discord> <r​pardini> could be, but is a risky move. 5.10's a mess organization wise, but is is stable-ish
14:42:56 <Armbian-Discord> <a​deepv> @rpardini what is broken in 5.18 but not in 5.19? 🙂
14:43:03 <Armbian-Discord> <M​anoftheSea> I'll try 5.19 on mvebu64
14:43:20 <Armbian-Discord> <r​pardini> oh wow adeepv literally everything we didn't fix in that PR 😉
14:43:21 <Armbian-Discord> <07I​gorPec> 5.10.y has that LDS problem with headers
14:43:28 <Armbian-Discord> <07I​gorPec> so DKMS fails
14:45:20 <Armbian-Discord> <a​deepv> @rpardini Yes, it turns out that all my devices works better on 5.16+ than on 5.10 anyway
14:45:30 <Armbian-Discord> <M​anoftheSea> Though ... Ebin works in Debian, too, and there are no armbian patches. I could see armbian dropping support
14:46:11 <Armbian-Discord> <r​pardini> adeepv: you're right. Khadas, Radxa, Jethub might work ok on 5.11-5.18 despite the shutdown revert caused by odroid. But odroid's themselves are severely borked
14:46:37 <IgorPec> teknoid_ i have around 40-50 boards in the rack just for testing purpose. But interface or options of using it are rather limited. If we could improve / expand that, it would be much easier
14:46:43 <Armbian-Discord> <08T​onymac32> ahh, hardkernel troubles
14:46:54 <Armbian-Discord> <08T​onymac32> mark unsupported, move on
14:46:57 <Armbian-Discord> <08T​onymac32> 😉
14:47:03 <Armbian-Discord> <r​pardini> hardkernel troubles not only fuck themselves, but also other meson64's
14:47:10 <IgorPec> the idea behind is that anyone from the team could use those devices anytime
14:47:59 <Armbian-Discord> <r​pardini> yeah... I've similar ideas... so a power control, plus UART, plus SD-mux would make every board remotely useful for testing
14:48:09 <Armbian-Discord> <r​pardini> but is a lot of HW to enable testing
14:48:18 <Armbian-Discord> <07I​gorPec> i have power control, some have uarts, sd mux never got to life but 😉
14:48:44 <Heisath> rpardini we have worked on all that.
14:48:50 <Armbian-Discord> <r​pardini> yeah, raise "wanted: SD-mux like solution for enabling testing" (dunno syntax for meeting script?)
14:48:53 <Heisath> But sd card muxing is just really hard.
14:49:24 <Heisath> https://github.com/armbian/mpads
14:49:25 <ArmbianHelper> ^ GitHub - armbian/mpads: Multiple power and data switch
14:49:25 <Armbian-Discord> <07I​gorPec> at least we tried
14:49:25 <Werner> #help wanted: SD-mux like solution for enabling testing
14:49:49 <Armbian-Discord> <08T​onymac32> lanefu found that single-SD card setup from Tizen via hackaday
14:49:54 <Heisath> I still have the boards around, if anybody wants to pick this up, I'd gladly share my experiences.
14:49:58 <Armbian-Discord> <08T​onymac32> we just need the epic usb hub of doom
14:50:04 <Armbian-Discord> <07I​gorPec> but regardless of this functionality - one step further would already be valuable
14:50:19 <Armbian-Discord> <07I​gorPec> like to give access to anyone to any of those boards
14:50:38 <Armbian-Discord> <07I​gorPec> so anyone can run manual update or whole CI
14:50:59 <Armbian-Discord> <07I​gorPec> or just plug everything to the main CI and test every time
14:50:59 <teknoid_> at least for me some SD extenders like this : https://www.adafruit.com/product/3688
14:51:00 <ArmbianHelper> ^ Micro SD Card Extender - 68cm (26 inch) long flex cable : ID 3688 : $6.95 : Adafruit Industries, Unique & fun DIY electronics and kits
14:51:00 <Armbian-Discord> <a​deepv> @IgorPec Without the ability to automatically restore the system from scratch it is useless
14:51:01 <Armbian-Discord> <r​pardini> very nice Heisath I'm very much interested
14:51:13 <Armbian-Discord> <08T​onymac32> I have the built boards from Olimex and could taka  look at some of it, it seems clear enough we need to have something on each device, star pattern is going to be a mess
14:51:51 <Armbian-Discord> <07I​gorPec> adeepv: well, we could boot from NFS and have rootfs there
14:51:55 <Armbian-Discord> <08T​onymac32> SD is not a robust long distance signal path  🙂
14:52:03 <[TheBug]> Igor: should tell your son you will double owance if he will sit in the room swapping sdcard for us for some weekends ;) (joking)
14:52:05 <Armbian-Discord> <07I​gorPec> so, its "just" boot loader that has to be tested manually
14:52:10 <[TheBug]> allowance*
14:52:20 <Armbian-Discord> <r​pardini> long flex cable for SD sounds like data loss waiting to happen, although I want it ♥️
14:52:22 <Armbian-Discord> <07I​gorPec> Yeah, i will swap SD cards at will 😉
14:52:23 <Armbian-Discord> <a​deepv> @IgorPec I'm slowly deleting uboot, your move?)
14:52:34 <Armbian-Discord> <07I​gorPec> heeh
14:52:43 <Armbian-Discord> <08T​onymac32> yeah, do they have those in 10 CM or so?
14:52:52 <Heisath> Even uboot does not need sd card swapping. Just override out of running linux. It only needs manual attention, if next boot fails :)
14:52:56 <vpeter> USB-SD-Mux from Linux Automation is really great. Expensive but great :)
14:53:34 <Armbian-Discord> <07I​gorPec> yes, i can buy one, but 40 is a bit too much 😉
14:53:42 <Armbian-Discord> <08T​onymac32> ah GmbH means money
14:53:49 <Armbian-Discord> <08T​onymac32> 😉
14:54:15 <Armbian-Discord> <08T​onymac32> this looks like the Tizen device rotated 45 degrees for style points
14:54:16 <vpeter> I did some similar device myself. Not computer controlled but still great if you don't need swapping sd cards.
14:54:18 <Armbian-Discord> <M​anoftheSea> It's 111 euro. I dunno what that means in freedom bucks, but it's more than my next sbc fix
14:54:20 <Heisath> but 111 per piece is only a few thousand total. Given how much you waste on support it might be worth it.
14:54:37 <Heisath> dollar euro is pretty much 1:1 right now.
14:54:47 <Armbian-Discord> <08T​onymac32> $140 unless the war continues, then maybe $50
14:54:52 <Armbian-Discord> <r​pardini> Heisath: true. It's what I am doing currently, A/B switching kernel/rootfs, but keep uboot in general. When going to test u-boot, have remote-hands at the ready for when it fails. Everything else is power control + UART
14:55:26 <Armbian-Discord> <r​pardini> Also USB-SD-Mux from Linux Automation is USB2-only, so very slow
14:55:30 <Armbian-Discord> <07I​gorPec> if / where cash is behind, its not a problem to buy this
14:55:43 <Armbian-Discord> <08T​onymac32> https://wiki.tizen.org/SDWire
14:55:45 <ArmbianHelper> ^ SDWire - Tizen Wiki
14:55:49 <Armbian-Discord> <07I​gorPec> but for community targets, if community do the testings, we are good
14:55:54 <Armbian-Discord> <a​deepv> @rpardini amlogic can usb-flash-boot when no data on emmc/sdcard
14:55:56 <Heisath> pardini good luck getting sd card mux with usb3 :D
14:56:24 <Armbian-Discord> <08T​onymac32> USB2 will be the best you get unless you design something yourself
14:56:28 <Armbian-Discord> <r​pardini> adeepv: true. also crazy HDMI i2c boot-chooser... (by Neil)
14:56:37 <Armbian-Discord> <08T​onymac32> I need to build that
14:56:39 <Armbian-Discord> <08T​onymac32> and this
14:57:16 <Armbian-Discord> <07I​gorPec> well, this tool on a side. we need a test system
14:57:23 <Armbian-Discord> <07I​gorPec> something better then current POC level
14:57:49 <Armbian-Discord> <r​pardini> yeah. getting test rig setup will be the key to stability -- I propose this needs its own comittee / initiative
14:58:02 <Armbian-Discord> <08T​onymac32> agreed
14:58:09 <Armbian-Discord> <r​pardini> eg, be vendor-driven and sponsored
14:58:56 <Armbian-Discord> <07I​gorPec> this is now under DevOps i think
14:59:11 <Armbian-Discord> <07I​gorPec> buy agree, this should be a separate entity
14:59:19 <[TheBug]> actually if accessible to our partners that may be a very good addition to the 'Armbian Professional' as a good valued for our Partners (vendors)
14:59:22 <Armbian-Discord> <r​pardini> it's too important to be buried under DevOps I think
14:59:36 <Armbian-Discord> <07I​gorPec> i tried to put it together, but with volunteers driving voolonteers ... its difficult
14:59:41 <Armbian-Discord> <r​pardini> yep, that actually adds a lot of value from vendor's perspective
14:59:57 <Armbian-Discord> <r​pardini> "actually tested releases on actual hardware"
15:00:06 <Armbian-Discord> <07I​gorPec> ok, then we will setup this commeetee, but who to put in? 🙂
15:00:20 <teknoid_> I need to leave, bye... happy releasing ;-)
15:00:23 <Armbian-Discord> <07I​gorPec> i can name people for R&D and devops easily, but not here
15:00:29 <[TheBug]> teknoid_: thanks for joining us
15:00:31 <Armbian-Discord> <r​pardini> lol, do all commitee's have all the same people (us) in them? hehe
15:00:33 <Armbian-Discord> <07I​gorPec> tehnoid: thanks !
15:00:43 <[TheBug]> before you all leave
15:00:46 <[TheBug]> let me say a quick thing on that
15:00:53 <Armbian-Discord> <07I​gorPec> we can divide people to devops and devs pretty well
15:01:02 <[TheBug]> hopefully the initially created committees and governance will start to be finalize over the next weeks
15:01:05 <Armbian-Discord> <07I​gorPec> or developers / support
15:01:30 <[TheBug]> this will have us do a new presentation as it is finalized and we will start to do a call to action for committee members to join our committees
15:01:49 <[TheBug]> rpardini: to answer your question, at start it may be like that but my hope is if people get involved that won't be the case over time
15:01:52 <Armbian-Discord> <07I​gorPec> but the key is what committee will be deciding upon, not who will be in or not
15:02:38 <Armbian-Discord> <07I​gorPec> now all questions and decisions are too mixed
15:03:20 <[TheBug]> If any of you have time to be on the one of the committees and has an interest, please feel free to reach out to me directly on irc/forum/discord and let me know so I can contact you as things are organized.
15:03:31 <Armbian-Discord> <r​pardini> seems like we need moar ppl -- but again, adding more people needs more people, ad infinitum
15:03:48 <Armbian-Discord> <M​anoftheSea> I, too, need to go. Mvebu64 (ebin) has no new problems, I'll test on 5.19 and provide feedback to move stable to it.
15:03:49 <Armbian-Discord> <07I​gorPec> and more complicated managemetn
15:04:13 <Armbian-Discord> <a​deepv> You can try to include me to DevOps. Maybe in half a year I'll have plenty of time 🙂
15:04:21 <[TheBug]> rpardini impressively we continue to add a few volunteers a week, even though it doesn't seem that way -- the bigger issue is attracting those who have more of the higher level developer skillsets and the time to volunteer.   Please do invite any developers you know to get involved!
15:04:26 <Armbian-Discord> <07I​gorPec> but more complicated management / introducing sections / comeetees  should help.
15:05:13 <Armbian-Discord> <r​pardini> yeah. from my side, I still wanna do a few "lecture" style videos
15:05:22 <Armbian-Discord> <r​pardini> to help onboard ppl
15:05:49 <[TheBug]> rpardini: then the best people to talk with are indeed Werner and NicoD as they have been running the video projects for Armbian
15:05:51 <Armbian-Discord> <07I​gorPec> i think we miss also the part - after they board
15:05:56 <Armbian-Discord> <r​pardini> "lets build Armbian 101"  "lets add a board which is already mainlined 101"
15:06:16 <[TheBug]> rpardini: I would encourage you to seriously reach out to them as I know they will be interested in helping you get that done
15:06:41 <Armbian-Discord> <07I​gorPec> how to answer "i want to help armbian" question and how to keep them motivated
15:06:48 <Werner> There are a couple of ideas already but nothing really pushed forward
15:06:53 <Armbian-Discord> <r​pardini> Yeah. I defninitely need to unload armbian-next, pff, what a fiasco
15:07:10 <Armbian-Discord> <r​pardini> really gotta cut the crap, remove what's not working, and release it
15:07:37 <Armbian-Discord> <07I​gorPec> we can do quick overview on  the situation if you got time?
15:08:02 <Armbian-Discord> <r​pardini> yeah I've still to consolidate a lot... but in the end that branch is 5-6 things in one or more
15:08:03 <Armbian-Discord> <07I​gorPec> regarding releases - do we have some solid conclusions?
15:08:24 <Werner> Discussion armbian-next is probably out of scope of this meeting though since we overshot time heavily already
15:08:24 <[TheBug]> IgorPec: I think actually the better idea is a sub-committee under R&D -- that may be what we want there in the short time for that extra committee you were mentioning -- as the R&D committee is put together we can discuss producing that sub-comitte as one of it's first meetings if that is of interest.
15:08:32 <Armbian-Discord> <r​pardini> I'd say we should move in direction of a release freeze / bugs only
15:08:51 <Armbian-Discord> <07I​gorPec> make a small and safe release
15:09:35 <Armbian-Discord> <r​pardini> well if 5.20 is 6.x... 5.19's release might be very important for many years
15:09:36 <Armbian-Discord> <07I​gorPec> releases of images are by default made from apt.armbian.com
15:09:43 <Armbian-Discord> <07I​gorPec> what its there, gets into the image
15:10:01 <Armbian-Discord> <07I​gorPec> so, i can "cherrypick", add to repository and run "make"
15:10:15 <Armbian-Discord> <07I​gorPec> just what to cherry pick, i have to get from you and maintainers
15:11:15 <Armbian-Discord> <07I​gorPec> this means i can only upgrade a few kernels, no u-boot, or some u-boot
15:13:20 <Armbian-Discord> <r​pardini> wait, I'm a bit confused
15:14:09 <Armbian-Discord> <r​pardini> "normal" release cycle would be to freeze master (bug fixes only) for a while, then create RC branch, unfreeze master... fix RC branch... then do a big build of all uboots/bsps/kernels, publish to repo, then build images
15:14:12 <Armbian-Discord> <r​pardini> or am I wrong?
15:14:33 <Armbian-Discord> <r​pardini> dunno how that coordinates with BETA=yes etc
15:14:34 <Armbian-Discord> <07I​gorPec> yes
15:14:49 <Armbian-Discord> <07I​gorPec> beta=yes means it will be published to beta repositoryy
15:14:59 <Armbian-Discord> <r​pardini> but can images be build from beta repo?
15:15:00 <Armbian-Discord> <07I​gorPec> beta=no main repo
15:15:05 <Armbian-Discord> <07I​gorPec> also
15:15:29 <Armbian-Discord> <r​pardini> because to me it seems that once we publish to non-beta repo, apt upgrade users will be affected, images having worked or not
15:15:53 <Armbian-Discord> <r​pardini> so it would be ideal to hold off on main-repo updates until we have confirmation that images build & work
15:15:59 <Armbian-Discord> <07I​gorPec> i did this last time. I build everything, pushed to beta and build stable images from beta repo
15:16:17 <Armbian-Discord> <07I​gorPec> then once happy, updated main repio
15:17:02 <Armbian-Discord> <a​deepv> @rpardini freezing the code just allows you to first make a beta, test and then build the main
15:17:38 <Armbian-Discord> <07I​gorPec> https://cdn.discordapp.com/attachments/857944478269702214/1008031585473396757/unknown.png
15:17:39 <ArmbianHelper> ^ [image/png] (29.9KiB)
15:18:30 <Armbian-Discord> <r​pardini> well I understand that driving that process is human-intensive
15:18:46 <Armbian-Discord> <07I​gorPec> yes, this is why we have this meeting 🙂
15:19:11 <Armbian-Discord> <r​pardini> also... it ends up being your work IgorPec always, even if we had a Release Manager person
15:19:18 <Armbian-Discord> <r​pardini> or am I misunderstanding?
15:19:34 <Armbian-Discord> <07I​gorPec> mainly it was my work, but i did get occasional help
15:19:54 <Armbian-Discord> <r​pardini> eg, suppose that builders might need manual cleaning, etc
15:19:55 <Armbian-Discord> <07I​gorPec> when i was not leading, i was helping
15:20:34 <Armbian-Discord> <07I​gorPec> many things has been automatized one the way, so most problematic is still testing and leading this whole process
15:21:16 <Armbian-Discord> <07I​gorPec> each release starts at least 1 month priour executing those buttons
15:21:30 <Armbian-Discord> <r​pardini> yeah. I'm afraid if I try to help there, I'll end up wanting to code-change everything and not help at all 😳
15:21:51 <Armbian-Discord> <07I​gorPec> haha
15:22:09 <Armbian-Discord> <07I​gorPec> code change on the way is addictive
15:23:32 <Armbian-Discord> <r​pardini> Yeah. I dunno what to propose for this release (22.08?)
15:23:38 <[TheBug]> circling back, what is the short term decision here -- will we make a 'major' or 'minor' release at the end of month or will we hold off till next release period?
15:24:14 <Armbian-Discord> <r​pardini> we need a release manager (not Igor, not TheBug) before anything can happen right?
15:24:26 <Armbian-Discord> <07I​gorPec> that would be ideal, yes
15:24:49 <Armbian-Discord> <r​pardini> also we need some kind of organized human testing too right?
15:25:05 <Armbian-Discord> <r​pardini> we could 30-day defer the release until we have atleast these 2 things
15:25:09 <[TheBug]> well good thing about that is I have been collecting contact information for all of the maintainers
15:25:22 <[TheBug]> so when we know we need testing we can reach out in a productive way and ask them to start the testing
15:25:26 <[TheBug]> and ask them to fill the forms
15:25:32 <Armbian-Discord> <07I​gorPec> we can do a small release for 22.08
15:25:43 <Armbian-Discord> <07I​gorPec> but it will only be tested via auto system
15:25:59 <Armbian-Discord> <07I​gorPec> without u-boot changes from 22.05 it can't do much damages
15:26:16 <Armbian-Discord> <07I​gorPec> image corruption test is automtic
15:26:25 <Armbian-Discord> <07I​gorPec> that wasn't in 22.05
15:26:53 <Armbian-Discord> <06T​heBug> @rpardini @Tonymac32  etc -- thoughts on minor release for 22.08 or questions? Do you disagree?
15:26:55 <Armbian-Discord> <r​pardini> image corruption test, is that the debsums MD5 check Igor?
15:27:13 <Armbian-Discord> <r​pardini> I'm not clear how a "minor release" is less work for Igor than a full release?
15:27:33 <Armbian-Discord> <07I​gorPec> that and compressed image test unpack
15:27:40 <Werner> I'd say skip release entirey. No need to force something just to have something
15:27:50 <Armbian-Discord> <07I​gorPec> we have OOMs in a few occasions
15:28:23 <Armbian-Discord> <r​pardini> I'd be sad not to release meson64's 5.19 as edge (at least)
15:28:42 <Armbian-Discord> <r​pardini> was a lot of work, and previous edge's were disappointing or borked
15:29:07 <Armbian-Discord> <07I​gorPec> we can release without kernel changes where things are questionable
15:29:33 <Armbian-Discord> <07I​gorPec> its like 22.08 "upgrade kernel for odroid xu4 and rockchip"
15:29:42 <Armbian-Discord> <07I​gorPec> we can do that for example
15:30:09 <Armbian-Discord> <r​pardini> you mean... you'd fork the 22.05 release branch as 22.08, and then just cherry-pick a few things here and there from master?
15:30:27 <Armbian-Discord> <r​pardini> I'd be against that.
15:30:46 <Armbian-Discord> <07I​gorPec> no
15:31:05 <Armbian-Discord> <07I​gorPec> build system is 22.08
15:31:14 <Armbian-Discord> <07I​gorPec> but repository is what it is
15:31:43 <Armbian-Discord> <07I​gorPec> with updated only kernels which are sane to update
15:31:50 <Armbian-Discord> <07I​gorPec> build from 22.08 of course
15:32:01 <Armbian-Discord> <07I​gorPec> this is kind of mini release
15:32:03 <Armbian-Discord> <r​pardini> hmm, I thought build train would build all kernels
15:32:06 <Armbian-Discord> <07I​gorPec> where most of the things remain the same
15:32:31 <Armbian-Discord> <07I​gorPec> it will build all of them, but i don't need to use them all
15:32:50 <Armbian-Discord> <r​pardini> you mean, it would build all, but you would only publish to repo a few of them?
15:32:52 <Armbian-Discord> <07I​gorPec> anyway this part is semi manual
15:33:02 <Armbian-Discord> <07I​gorPec> yes
15:33:05 <Armbian-Discord> <07I​gorPec> that
15:33:09 <Armbian-Discord> <r​pardini> ahn ok took me a while, sorry
15:33:21 <Armbian-Discord> <07I​gorPec> this is some sane middle way to me
15:33:34 <Armbian-Discord> <07I​gorPec> a compromise
15:34:09 <Armbian-Discord> <r​pardini> yeah. I'd say fire off the train, lets see what builds and what not. Also, what builds cleanly (all patches applying green/ok)
15:34:23 <Armbian-Discord> <07I​gorPec> everything builds, that's not a question
15:34:32 <Armbian-Discord> <r​pardini> ahn. interesting.
15:34:48 <Armbian-Discord> <r​pardini> we'd need at least to fix/hardcoded the tag:xxx for every family in the release branch
15:35:04 <Armbian-Discord> <r​pardini> although sunxi's are already that way in master it looks like, others aren't
15:36:00 <Armbian-Discord> <r​pardini> also, might be a good rule of thumb for "minor releases" to NOT change any current kernel, only edge (except for bugfixes on current maybe)
15:36:16 <Armbian-Discord> <07I​gorPec> yes, that is a lot more predicatble
15:36:42 <Armbian-Discord> <r​pardini> also that makes it easier for you: build train, then rm *current*, then publish 😉
15:36:43 <Armbian-Discord> <07I​gorPec> something we can release with smoke tests only
15:37:35 <Armbian-Discord> <07I​gorPec> that could be doable
15:37:45 <Armbian-Discord> <a​deepv> am I correct in assuming that minor releases allow you to change everything except kernel/uboot patches?
15:38:00 <Armbian-Discord> <07I​gorPec> probably a great solution. we provide everything others do, but keep stability at least the same
15:38:22 <Armbian-Discord> <r​pardini> or maybe cat targets.conf | grep -v current
15:38:26 <Armbian-Discord> <07I​gorPec> yes
15:38:28 <Armbian-Discord> <a​deepv> how and where to save changes for the kernel/uboot. are we not just going to wait a quarter for a minor release to make changes?
15:39:58 <Armbian-Discord> <07I​gorPec> we will probably making a pro version with "update at anytime" support
15:40:34 <Armbian-Discord> <07I​gorPec> this brings up too many problems
15:41:57 <Armbian-Discord> <r​pardini> yep. for pro version, we need some serious repository management abilities, maybe even a full repo per-version, not only main/beta
15:42:11 <Armbian-Discord> <r​pardini> eg: who knows the codename for 22.05? (I don't.)
15:42:14 <Armbian-Discord> <07I​gorPec> regarding u-boot and updating, u-boot packages are still per branch so this is not a problem
15:42:31 <Armbian-Discord> <07I​gorPec> if we skip CURRENT, others will still be update
15:42:59 <Armbian-Discord> <07I​gorPec> 22.08    Yapok
15:43:16 <Armbian-Discord> <r​pardini> yeah. that sounds like a good compromise for now. I'll try: for 22.08, we're doing a minor release of edge stuff only. current (and legacy etc) will not be built or updated in this release
15:43:38 <Armbian-Discord> <07I​gorPec> legacy actually can
15:44:38 <Armbian-Discord> <r​pardini> yeah, can, but I'd stick to edge to enforce the mainline / community aspect of it
15:44:50 <Armbian-Discord> <07I​gorPec> ok, fine for me
15:45:27 <Armbian-Discord> <07I​gorPec> then we don't need testers for this one 😉 Or lets say this is optional
15:45:41 <Armbian-Discord> <07I​gorPec> as EDGE is anyway ... EDGE
15:47:02 <Armbian-Discord> <07I​gorPec> and the same goes for 2nd next, 23.02 release
15:47:03 <Armbian-Discord> <r​pardini> well I'll test meson64 odroid stuff ofc I also have rockpro64, quartz64a, rpi4b available out of production to test
15:47:31 <Armbian-Discord> <r​pardini> and the damned tinkerboard-2 😉
15:47:52 <Werner> To cut a long story short (and come to an end of the half hour meeting ^^) there will be a release with minor stuff, correct?
15:48:00 <Armbian-Discord> <07I​gorPec> yes,
15:48:04 <Heisath> Yeah we are running in circles.
15:48:11 <jock> the checklist here https://www.armbian.com/rc-testing/ requires to be filled anyway for this mini-release ? I think it is a good idea to do that anyway...
15:48:12 <ArmbianHelper> ^ RC Testing – Armbian
15:48:19 <Werner> #agreed Release yes. Minor changes only
15:48:27 <Heisath> Yes fill out checklist.
15:48:29 <Armbian-Discord> <07I​gorPec> EDGE kernel upgrade only
15:48:49 <Werner> So anything left that should be included in the meeting protocol?
15:48:51 <Armbian-Discord> <07I​gorPec> but full images rebuild from current build branch
15:49:19 <Armbian-Discord> <07I​gorPec> that second next release will be the same, so two minor, two major releases per year
15:49:42 <Werner> #agreed  <I​gorPec> that second next release will be the same, so two minor, two major releases per year
15:50:44 <Armbian-Discord> <07I​gorPec> as far as release meeting topic, we are done. a bit over schedule as always 😉
15:50:51 <Werner> yup
15:50:53 <[TheBug]> #agreed Fill out rc-testing checklist as part of mini-release
15:50:56 <Armbian-Discord> <a​deepv> guys, can i skip filling in https://www.armbian.com/rc-testing/? jethome has its own path with test/use release.
15:50:57 <ArmbianHelper> ^ RC Testing – Armbian
15:51:55 <Werner> Aight
15:51:57 <Werner> #endmeeting