14:00:56 #startmeeting Release meeting 22.08 14:00:56 Meeting started Sat Aug 13 14:00:56 2022 UTC. The chair is Werner. Information about MeetBot at http://wiki.debian.org/MeetBot. 14:00:56 Useful Commands: #action #agreed #help #info #idea #link #topic. 14:01:08 #topic check-in 14:01:14 Here. 14:01:22 #topic check-in 14:01:23 hello 14:01:47 Hello! 14:02:17 <08T​onymac32> hello 14:02:21 <[TheBug]> Hey rpardini! 14:02:26 <[TheBug]> Hey teknoid_! 14:02:31 Also hello here 14:02:51 here) 14:02:53 Does it read people in discord? 14:03:03 it does 14:03:04 <[TheBug]> yep there is bridge 14:03:30 Change from May? 14:03:42 <[TheBug]> Wait a few more minutes for straglers then @IgorPec can kick things off :) 14:04:20 changes are always huge or a lot bigger than we can handle them 14:04:28 but that is not the only problem we have 14:04:39 <08T​onymac32> yep, we're changing to all vendor kernels 14:04:42 <08T​onymac32> the older the better 14:04:47 I mean the bridge didn't read me in May 14:05:12 <[TheBug]> BTW Thanks to everyone who has taken time out of their weekend to be here and help support Armbian. I just want you to know that we appriciate you all and you time and dedication to Armbian! 14:05:21 we have to think over the whole release model principle 14:06:02 if we want to proceede with the same level (well tested release, well prepared release) ... the 14:06:11 then this means serious work 14:06:23 I've just rebased 3-4 weeks of master onto -next. It's mostly moving edge's to 5.19, some new vendor-kernel-based boards 14:06:49 yes, we have a lot of happening, but just a few days in total to do something. 14:06:55 lotsa rootfs cache changes, otherwise more of the same stuff. some board-side scripts might need a lot of testing 14:07:33 but I've a feeling we could focus testing on 1 or 2 boards per-family and have a good general indication 14:07:33 well, from developers perspective we have "good stability" 14:07:42 #topic release preparation and discussion 14:07:56 my two boards I support are in productive use - for me its hard to do full image testing 4 times per year as they are built into enclosures 14:08:10 board count is under better control since last year, but board count solo doesn't help much 14:08:16 testing kernel is no problem, but full image is a big problem 14:08:26 for testing kernels we do have automation 14:08:31 teknoid get more boards then. Testing in production is shit. 14:08:33 <08T​onymac32> uboot often brings challenges and must be tested fresh 14:08:36 but is not well maintained 14:08:47 yep, u-boot 14:08:58 idea is to move many many boards to most recent u-boot 14:09:02 these boards are out of production for years ;-) 14:09:10 I haven't seen meson64 u-boot changes. I guess ya'll mean rockchip's u-boot? 14:09:25 just beeing sure that this goes as expected is complex job 14:09:28 (mostly... of course) 14:09:32 <08T​onymac32> just in general. Testing a new image by just the kernel misses everything 14:09:44 meson is pretty data i think 14:09:49 oh indeed 14:09:51 <08T​onymac32> I've been guilty of that and got burned 14:09:56 but we have allwinner and rockchip which is old 14:10:11 also it would be sane to move to EFI in order to use some standard tools 14:10:22 rpardini i think we can just move meson64 uboot to 2022.07 14:10:22 <08T​onymac32> allwinner needs bumped, I can voich for GX/GXL, but the newer stuff is... 14:10:31 <08T​onymac32> vouch* 14:10:36 adeepv: indeed. I've that feeling as well 14:10:42 <08T​onymac32> amlgoic. #morecoffee 14:10:43 yes, just the question is when and how to test? 14:11:24 project "Regular release" has to be well prepared and lead. Neither me or TheBug can pick this up in "by the way" 14:11:54 yeah. I suppose when developers _also_ have to test we get bogged down very quickly 14:11:55 are 4 full stream releases too much perhaps? 14:12:15 testings is getting organised better, we do have more voolonteers for this job 14:12:20 my proposal is to align releases with debian / ubuntu release cycle 14:12:23 Moving to EFI means u-boot supporting EFI, right? Seems beyond armbian control 14:12:34 <08T​onymac32> u-boot does support EFI 14:12:38 but again, someone needs to lead them. Currently Bug is trying to cover that to some degree 14:12:56 <[TheBug]> Yes, we have been looking for someone to take on this task, in fact we are discussing trying to come up with enough money for a stipend to give to the 'Release Manager' and maybe even try to keep a single manager for more than one release if they are good with the job. If this is something any of you would have an interest in specifically, please reach out to me directly after the meeting! 14:13:09 Regarding the 4 release question from Igor: IMHO yes. Kernel is kind a rolling anyways and two month until next freeze...barely anything can be done within that time window 14:13:29 IgorPec Maybe two releases per year would be better? 14:13:37 Agree. I second 2 releases. 14:13:44 we heard many ideas, like 2 big, 2 small 14:13:56 small = like no kernel bump 14:14:06 I think we need one release that makes things really stable first... then we can move to 2/year 14:14:07 staying on the same version, just patch bump 14:14:24 <[TheBug]> Yes I am not sure we really came up with a schedule for 'minor releases' or what that actually means, but maybe if we can define this we could make more 'minor' releases and less 'major' releases? 14:14:39 really stable can only be achived with massively improved testings . to at least know what is wrong 14:14:55 in the end the release cadence should be dictated by our testing frequency / trustworthiness 14:15:02 but currently testings is - down to developers mainly 14:15:44 <[TheBug]> BTW https://www.armbian.com/rc-testing/ is our first attempt at being able to track testing -- if you guys find that there could be improvements here, please communicate them to me so I can see if I can continue to improve this form / process 14:15:45 ^ RC Testing – Armbian 14:16:35 <[TheBug]> This is also important because it will submit all of these testing into a single Jira ticeket which all can view including release manager: https://armbian.atlassian.net/browse/AR-1271 14:16:36 4AR-1271 6[Task] "RC Testing for August 2022 Release" reported by 3TheLinuxBug at 2022-07-27. Status: To Do 14:16:37 ^ [AR-1271] - Jira 14:16:42 also there are ideas to have two repositories 14:16:44 very interesting TheBug. Maybe worth it to do a on-board script that reports stuff like armbianmonitor and the answers to test questions 14:16:54 one rolling community, basically current trunk 14:17:19 move everything there and have pro grade branch with a lot more testings 14:17:25 IgorPec: makes sense. since we have 0 testing capacity, nightly-only always-dirty releases are all we can afford 14:17:28 <[TheBug]> (please note the ticket number will change every release in case you were planning to bookmark that) 14:17:59 <[TheBug]> rpardini: if you would be interested in helping design that and come up with an easy method for submission of the information I am fully on board with that idea 14:18:28 basic testing capacity is this https://github.com/armbian/scripts/actions/runs/2544689819 14:18:29 ^ Smoke tests · armbian/scripts@8942200 · GitHub 14:19:10 TheBug: yes. Count me in. (goes together with my build-log uploader, firstrun-rewrite, and other initiatives) 14:19:35 <[TheBug]> rpardini: excelent. We will discuss this more in the near future then. 14:19:43 but before developing this further, I think "release manager" is quite a serious role here 14:19:47 <[TheBug]> indeed. 14:20:04 yeah, I've never been release manager, and have no idea what it really entails 14:20:17 it means followint the agenda and talking to people 14:20:40 we do have agenda, perhaps its not the best, but it gets us where we want 14:20:40 <[TheBug]> Also if you guys don't feel like any of you would have time, but know someone who could fit the role looking for some part-time work that would take it serious, please send them our way. It doesn't have to be one of y ou take on this but we do need someone who has enough skillsets to work with everyone and help to make the release happen. 14:21:03 what material do we have? agenda? 14:21:08 https://docs.armbian.com/Process_Release-Model/ 14:21:09 ^ Model - Armbian Documentation 14:21:29 https://docs.armbian.com/Process_Release-Model/#release-coordinating 14:21:30 ^ Model - Armbian Documentation 14:21:52 it might need changes and improvements 14:22:19 Ok but that is already a very good start 14:22:31 <[TheBug]> And if someone wants to take on the role I will work along with them for the first release and we can work to improve documentation/procedures as that goes. SO you don't have to be concerned about being all alone 14:22:40 My question was more on the technical side. For example, the RC branch should have `tag:xx` instead of `branch:xxx` right? 14:22:49 yes, "just" we do not have people that would cover that. but we have a lot of people that would want "stability" 14:23:35 currently we branched whole build system and build from there 14:24:01 also, I've this in the back of my head, we need some heads-up/buy-in/agreement about the release from `media`, `sunxi` ppl (mostly in Russia I guess). How to communicate with them? 14:24:12 and those branches stayed there. Since sources were frozen before, in theory, its possible to remake the same image later on. 14:25:06 being able to "freeze" what fetch_from_repo pulls, based on what is the RC/release branch, is a pending feature on my (build system) side 14:25:13 "well, maintainers has to participate in the release process or board goes out" 14:25:34 <[TheBug]> rpardini: there was some conversation on this last night in fact between Tony, Igor and I, I know also jock has some of these concerns. I would tell you guys this should be first item in governance, but I think we may need to work on this before that can be implemented. As such we may want to setup a quick call together to discuss how that can done and invite Oleg as well so we can find 14:25:34 <[TheBug]> some comprimise. 14:25:39 this is a bit light as we also can't provide what we would like 14:26:41 but for release process, this is not that critical. Release is "hey folks, lets stabilize this huge system best way possible" 14:26:53 this is common work per se, but has to be coordinated 14:27:06 works for me 14:27:11 yeah. again, in the end, most of the pain points can/should be addressed by features in build system. except, of course, getting actual reproducible tests and results done by people who know what they're doing. 14:27:12 to get maximum outcome 14:28:02 build system adds its own problems here, but for releasing a distribution, its "just" a tool. which will break down too :) 14:28:24 yep. repo management for example is something 100% out of my reach 14:29:02 well, this is a complex system and its difficult to get things under any control 14:29:03 also, testing a regular user doing `apt update & apt dist-upgrade` is a much more complicated beast than testing `burn image to SD and it works` 14:29:27 apt update and upgrade is automatised, but on a clean system 14:29:54 that could also be something to me moved into `community` vs `pro` releases. community would not assure apt-distupgrade upgradeability 14:30:03 while pro might 14:30:27 <08T​onymac32> $$ 14:30:30 in general. levels of testing should greatly be improved. i am trying and trying but not much interest from general public to hold up into CI automation 14:31:19 yes, certain things will be moved under pro versions which are interested that thins are in better shape 14:31:30 yeah. I need a decent SD-mux like thingy to work that 14:31:48 other solutions won't reproduce user's experience like switching SD cards would 14:31:51 or if we would have ability to find sponsors for releases? 14:32:25 time lost is simply too big if you want to do it right 14:32:36 (eg: I've u-boot based A/B switching working for most bootscript-based boards... but that won't test uboot itself, only from kernel down) 14:32:55 <[TheBug]> rpardini: I think one thing we need to do somewhere is start to document tools which would improve the development process for our developers and maintainers somewhere and as Armbian can fund it possibly try and help provide those things? 14:33:19 we can setup a proper lava test system, but we are talking about a serious operation 14:34:23 so to recap. what we can do with what we have? 14:35:34 rpardini I am also thinking about how to automate testing for uboot, but I have no good ideas yet 14:35:36 what is easy is = build images, test them for corruption, run apt upgrade on 50+ boards 14:35:59 but this does not label release as "stable" nor "tested" 14:36:53 @IgorPec There is another thing: there are changes between releases that are not available by simply updating packages 14:37:38 <07I​gorPec> yes, i understand that. (automated) images testing would be perfect. 14:37:58 <07I​gorPec> now, we have more voolonteers that can help testing, but that represent us even more burden 14:38:09 <07I​gorPec> as someone has to talk with them, encorage them 14:38:39 <07I​gorPec> and we need to thank them 14:39:03 yep. I can propose to do myself 1 or at most 2 rounds of testing on N2, HC4, VIM3L 14:39:20 but, only from new-image-to-SD-and-see-if-it-works-for-me style 14:39:56 and that's only because I really wanna release 5.19 edge on meson64 😉 14:40:23 <[TheBug]> rpardini: as you have Vim3L maybe you can ping NicoD at some point this coming week and just chat with him on it, I know he is also maintaining and has been trying to help overcome some issues -- Tonymac32 and Tenkawa helped him a bit on previous weeks voice chat, but maybe you could check with him if there are any fixes you could help him get put in place for that board. 14:40:25 <08T​onymac32> is 5.19 an LTS or not? It does bring in a lot of good stuff 14:40:35 <07I​gorPec> 5.19.y as CURRENT? 14:40:47 great question, dunno about LTS status of 5.19, or if 5.20 is coming or not 14:40:53 5.15 is LTS 14:40:58 <07I​gorPec> probably 6.y is next 14:41:01 I guess 5.18 is too early for current 14:41:15 <08T​onymac32> I was part of the "Current should always be LTS" crowd 14:41:20 sorry, meant 5.19 14:41:21 <07I​gorPec> hmm, but with meson64 we have 5.10 for CURRENT 14:41:28 <07I​gorPec> which is not very goo 14:41:38 yeah, but 5.19 edge is super-new 14:41:44 <08T​onymac32> yeah that got missed, I think 5.15 was causing troubles with the newer boards 14:41:50 5.20 is 6.0 14:41:57 5.15 meson64 is completely borked 14:42:05 5.11 - 5.18 meson64 is completely borked btw 14:42:18 <07I​gorPec> than its better to move meson64-CURRENT directly to 5.19.y ? 14:42:22 <07I​gorPec> right now 14:42:25 I'll give 5.19 a try on my C2 next week 14:42:38 could be, but is a risky move. 5.10's a mess organization wise, but is is stable-ish 14:42:56 @rpardini what is broken in 5.18 but not in 5.19? 🙂 14:43:03 I'll try 5.19 on mvebu64 14:43:20 oh wow adeepv literally everything we didn't fix in that PR 😉 14:43:21 <07I​gorPec> 5.10.y has that LDS problem with headers 14:43:28 <07I​gorPec> so DKMS fails 14:45:20 @rpardini Yes, it turns out that all my devices works better on 5.16+ than on 5.10 anyway 14:45:30 Though ... Ebin works in Debian, too, and there are no armbian patches. I could see armbian dropping support 14:46:11 adeepv: you're right. Khadas, Radxa, Jethub might work ok on 5.11-5.18 despite the shutdown revert caused by odroid. But odroid's themselves are severely borked 14:46:37 teknoid_ i have around 40-50 boards in the rack just for testing purpose. But interface or options of using it are rather limited. If we could improve / expand that, it would be much easier 14:46:43 <08T​onymac32> ahh, hardkernel troubles 14:46:54 <08T​onymac32> mark unsupported, move on 14:46:57 <08T​onymac32> 😉 14:47:03 hardkernel troubles not only fuck themselves, but also other meson64's 14:47:10 the idea behind is that anyone from the team could use those devices anytime 14:47:59 yeah... I've similar ideas... so a power control, plus UART, plus SD-mux would make every board remotely useful for testing 14:48:09 but is a lot of HW to enable testing 14:48:18 <07I​gorPec> i have power control, some have uarts, sd mux never got to life but 😉 14:48:44 rpardini we have worked on all that. 14:48:50 yeah, raise "wanted: SD-mux like solution for enabling testing" (dunno syntax for meeting script?) 14:48:53 But sd card muxing is just really hard. 14:49:24 https://github.com/armbian/mpads 14:49:25 ^ GitHub - armbian/mpads: Multiple power and data switch 14:49:25 <07I​gorPec> at least we tried 14:49:25 #help wanted: SD-mux like solution for enabling testing 14:49:49 <08T​onymac32> lanefu found that single-SD card setup from Tizen via hackaday 14:49:54 I still have the boards around, if anybody wants to pick this up, I'd gladly share my experiences. 14:49:58 <08T​onymac32> we just need the epic usb hub of doom 14:50:04 <07I​gorPec> but regardless of this functionality - one step further would already be valuable 14:50:19 <07I​gorPec> like to give access to anyone to any of those boards 14:50:38 <07I​gorPec> so anyone can run manual update or whole CI 14:50:59 <07I​gorPec> or just plug everything to the main CI and test every time 14:50:59 at least for me some SD extenders like this : https://www.adafruit.com/product/3688 14:51:00 ^ Micro SD Card Extender - 68cm (26 inch) long flex cable : ID 3688 : $6.95 : Adafruit Industries, Unique & fun DIY electronics and kits 14:51:00 @IgorPec Without the ability to automatically restore the system from scratch it is useless 14:51:01 very nice Heisath I'm very much interested 14:51:13 <08T​onymac32> I have the built boards from Olimex and could taka look at some of it, it seems clear enough we need to have something on each device, star pattern is going to be a mess 14:51:51 <07I​gorPec> adeepv: well, we could boot from NFS and have rootfs there 14:51:55 <08T​onymac32> SD is not a robust long distance signal path 🙂 14:52:03 <[TheBug]> Igor: should tell your son you will double owance if he will sit in the room swapping sdcard for us for some weekends ;) (joking) 14:52:05 <07I​gorPec> so, its "just" boot loader that has to be tested manually 14:52:10 <[TheBug]> allowance* 14:52:20 long flex cable for SD sounds like data loss waiting to happen, although I want it ♥️ 14:52:22 <07I​gorPec> Yeah, i will swap SD cards at will 😉 14:52:23 @IgorPec I'm slowly deleting uboot, your move?) 14:52:34 <07I​gorPec> heeh 14:52:43 <08T​onymac32> yeah, do they have those in 10 CM or so? 14:52:52 Even uboot does not need sd card swapping. Just override out of running linux. It only needs manual attention, if next boot fails :) 14:52:56 USB-SD-Mux from Linux Automation is really great. Expensive but great :) 14:53:34 <07I​gorPec> yes, i can buy one, but 40 is a bit too much 😉 14:53:42 <08T​onymac32> ah GmbH means money 14:53:49 <08T​onymac32> 😉 14:54:15 <08T​onymac32> this looks like the Tizen device rotated 45 degrees for style points 14:54:16 I did some similar device myself. Not computer controlled but still great if you don't need swapping sd cards. 14:54:18 It's 111 euro. I dunno what that means in freedom bucks, but it's more than my next sbc fix 14:54:20 but 111 per piece is only a few thousand total. Given how much you waste on support it might be worth it. 14:54:37 dollar euro is pretty much 1:1 right now. 14:54:47 <08T​onymac32> $140 unless the war continues, then maybe $50 14:54:52 Heisath: true. It's what I am doing currently, A/B switching kernel/rootfs, but keep uboot in general. When going to test u-boot, have remote-hands at the ready for when it fails. Everything else is power control + UART 14:55:26 Also USB-SD-Mux from Linux Automation is USB2-only, so very slow 14:55:30 <07I​gorPec> if / where cash is behind, its not a problem to buy this 14:55:43 <08T​onymac32> https://wiki.tizen.org/SDWire 14:55:45 ^ SDWire - Tizen Wiki 14:55:49 <07I​gorPec> but for community targets, if community do the testings, we are good 14:55:54 @rpardini amlogic can usb-flash-boot when no data on emmc/sdcard 14:55:56 pardini good luck getting sd card mux with usb3 :D 14:56:24 <08T​onymac32> USB2 will be the best you get unless you design something yourself 14:56:28 adeepv: true. also crazy HDMI i2c boot-chooser... (by Neil) 14:56:37 <08T​onymac32> I need to build that 14:56:39 <08T​onymac32> and this 14:57:16 <07I​gorPec> well, this tool on a side. we need a test system 14:57:23 <07I​gorPec> something better then current POC level 14:57:49 yeah. getting test rig setup will be the key to stability -- I propose this needs its own comittee / initiative 14:58:02 <08T​onymac32> agreed 14:58:09 eg, be vendor-driven and sponsored 14:58:56 <07I​gorPec> this is now under DevOps i think 14:59:11 <07I​gorPec> buy agree, this should be a separate entity 14:59:19 <[TheBug]> actually if accessible to our partners that may be a very good addition to the 'Armbian Professional' as a good valued for our Partners (vendors) 14:59:22 it's too important to be buried under DevOps I think 14:59:36 <07I​gorPec> i tried to put it together, but with volunteers driving voolonteers ... its difficult 14:59:41 yep, that actually adds a lot of value from vendor's perspective 14:59:57 "actually tested releases on actual hardware" 15:00:06 <07I​gorPec> ok, then we will setup this commeetee, but who to put in? 🙂 15:00:20 I need to leave, bye... happy releasing ;-) 15:00:23 <07I​gorPec> i can name people for R&D and devops easily, but not here 15:00:29 <[TheBug]> teknoid_: thanks for joining us 15:00:31 lol, do all commitee's have all the same people (us) in them? hehe 15:00:33 <07I​gorPec> tehnoid: thanks ! 15:00:43 <[TheBug]> before you all leave 15:00:46 <[TheBug]> let me say a quick thing on that 15:00:53 <07I​gorPec> we can divide people to devops and devs pretty well 15:01:02 <[TheBug]> hopefully the initially created committees and governance will start to be finalize over the next weeks 15:01:05 <07I​gorPec> or developers / support 15:01:30 <[TheBug]> this will have us do a new presentation as it is finalized and we will start to do a call to action for committee members to join our committees 15:01:49 <[TheBug]> rpardini: to answer your question, at start it may be like that but my hope is if people get involved that won't be the case over time 15:01:52 <07I​gorPec> but the key is what committee will be deciding upon, not who will be in or not 15:02:38 <07I​gorPec> now all questions and decisions are too mixed 15:03:20 <[TheBug]> If any of you have time to be on the one of the committees and has an interest, please feel free to reach out to me directly on irc/forum/discord and let me know so I can contact you as things are organized. 15:03:31 seems like we need moar ppl -- but again, adding more people needs more people, ad infinitum 15:03:48 I, too, need to go. Mvebu64 (ebin) has no new problems, I'll test on 5.19 and provide feedback to move stable to it. 15:03:49 <07I​gorPec> and more complicated managemetn 15:04:13 You can try to include me to DevOps. Maybe in half a year I'll have plenty of time 🙂 15:04:21 <[TheBug]> rpardini impressively we continue to add a few volunteers a week, even though it doesn't seem that way -- the bigger issue is attracting those who have more of the higher level developer skillsets and the time to volunteer. Please do invite any developers you know to get involved! 15:04:26 <07I​gorPec> but more complicated management / introducing sections / comeetees should help. 15:05:13 yeah. from my side, I still wanna do a few "lecture" style videos 15:05:22 to help onboard ppl 15:05:49 <[TheBug]> rpardini: then the best people to talk with are indeed Werner and NicoD as they have been running the video projects for Armbian 15:05:51 <07I​gorPec> i think we miss also the part - after they board 15:05:56 "lets build Armbian 101" "lets add a board which is already mainlined 101" 15:06:16 <[TheBug]> rpardini: I would encourage you to seriously reach out to them as I know they will be interested in helping you get that done 15:06:41 <07I​gorPec> how to answer "i want to help armbian" question and how to keep them motivated 15:06:48 There are a couple of ideas already but nothing really pushed forward 15:06:53 Yeah. I defninitely need to unload armbian-next, pff, what a fiasco 15:07:10 really gotta cut the crap, remove what's not working, and release it 15:07:37 <07I​gorPec> we can do quick overview on the situation if you got time? 15:08:02 yeah I've still to consolidate a lot... but in the end that branch is 5-6 things in one or more 15:08:03 <07I​gorPec> regarding releases - do we have some solid conclusions? 15:08:24 Discussion armbian-next is probably out of scope of this meeting though since we overshot time heavily already 15:08:24 <[TheBug]> IgorPec: I think actually the better idea is a sub-committee under R&D -- that may be what we want there in the short time for that extra committee you were mentioning -- as the R&D committee is put together we can discuss producing that sub-comitte as one of it's first meetings if that is of interest. 15:08:32 I'd say we should move in direction of a release freeze / bugs only 15:08:51 <07I​gorPec> make a small and safe release 15:09:35 well if 5.20 is 6.x... 5.19's release might be very important for many years 15:09:36 <07I​gorPec> releases of images are by default made from apt.armbian.com 15:09:43 <07I​gorPec> what its there, gets into the image 15:10:01 <07I​gorPec> so, i can "cherrypick", add to repository and run "make" 15:10:15 <07I​gorPec> just what to cherry pick, i have to get from you and maintainers 15:11:15 <07I​gorPec> this means i can only upgrade a few kernels, no u-boot, or some u-boot 15:13:20 wait, I'm a bit confused 15:14:09 "normal" release cycle would be to freeze master (bug fixes only) for a while, then create RC branch, unfreeze master... fix RC branch... then do a big build of all uboots/bsps/kernels, publish to repo, then build images 15:14:12 or am I wrong? 15:14:33 dunno how that coordinates with BETA=yes etc 15:14:34 <07I​gorPec> yes 15:14:49 <07I​gorPec> beta=yes means it will be published to beta repositoryy 15:14:59 but can images be build from beta repo? 15:15:00 <07I​gorPec> beta=no main repo 15:15:05 <07I​gorPec> also 15:15:29 because to me it seems that once we publish to non-beta repo, apt upgrade users will be affected, images having worked or not 15:15:53 so it would be ideal to hold off on main-repo updates until we have confirmation that images build & work 15:15:59 <07I​gorPec> i did this last time. I build everything, pushed to beta and build stable images from beta repo 15:16:17 <07I​gorPec> then once happy, updated main repio 15:17:02 @rpardini freezing the code just allows you to first make a beta, test and then build the main 15:17:38 <07I​gorPec> https://cdn.discordapp.com/attachments/857944478269702214/1008031585473396757/unknown.png 15:17:39 ^ [image/png] (29.9KiB) 15:18:30 well I understand that driving that process is human-intensive 15:18:46 <07I​gorPec> yes, this is why we have this meeting 🙂 15:19:11 also... it ends up being your work IgorPec always, even if we had a Release Manager person 15:19:18 or am I misunderstanding? 15:19:34 <07I​gorPec> mainly it was my work, but i did get occasional help 15:19:54 eg, suppose that builders might need manual cleaning, etc 15:19:55 <07I​gorPec> when i was not leading, i was helping 15:20:34 <07I​gorPec> many things has been automatized one the way, so most problematic is still testing and leading this whole process 15:21:16 <07I​gorPec> each release starts at least 1 month priour executing those buttons 15:21:30 yeah. I'm afraid if I try to help there, I'll end up wanting to code-change everything and not help at all 😳 15:21:51 <07I​gorPec> haha 15:22:09 <07I​gorPec> code change on the way is addictive 15:23:32 Yeah. I dunno what to propose for this release (22.08?) 15:23:38 <[TheBug]> circling back, what is the short term decision here -- will we make a 'major' or 'minor' release at the end of month or will we hold off till next release period? 15:24:14 we need a release manager (not Igor, not TheBug) before anything can happen right? 15:24:26 <07I​gorPec> that would be ideal, yes 15:24:49 also we need some kind of organized human testing too right? 15:25:05 we could 30-day defer the release until we have atleast these 2 things 15:25:09 <[TheBug]> well good thing about that is I have been collecting contact information for all of the maintainers 15:25:22 <[TheBug]> so when we know we need testing we can reach out in a productive way and ask them to start the testing 15:25:26 <[TheBug]> and ask them to fill the forms 15:25:32 <07I​gorPec> we can do a small release for 22.08 15:25:43 <07I​gorPec> but it will only be tested via auto system 15:25:59 <07I​gorPec> without u-boot changes from 22.05 it can't do much damages 15:26:16 <07I​gorPec> image corruption test is automtic 15:26:25 <07I​gorPec> that wasn't in 22.05 15:26:53 <06T​heBug> @rpardini @Tonymac32 etc -- thoughts on minor release for 22.08 or questions? Do you disagree? 15:26:55 image corruption test, is that the debsums MD5 check Igor? 15:27:13 I'm not clear how a "minor release" is less work for Igor than a full release? 15:27:33 <07I​gorPec> that and compressed image test unpack 15:27:40 I'd say skip release entirey. No need to force something just to have something 15:27:50 <07I​gorPec> we have OOMs in a few occasions 15:28:23 I'd be sad not to release meson64's 5.19 as edge (at least) 15:28:42 was a lot of work, and previous edge's were disappointing or borked 15:29:07 <07I​gorPec> we can release without kernel changes where things are questionable 15:29:33 <07I​gorPec> its like 22.08 "upgrade kernel for odroid xu4 and rockchip" 15:29:42 <07I​gorPec> we can do that for example 15:30:09 you mean... you'd fork the 22.05 release branch as 22.08, and then just cherry-pick a few things here and there from master? 15:30:27 I'd be against that. 15:30:46 <07I​gorPec> no 15:31:05 <07I​gorPec> build system is 22.08 15:31:14 <07I​gorPec> but repository is what it is 15:31:43 <07I​gorPec> with updated only kernels which are sane to update 15:31:50 <07I​gorPec> build from 22.08 of course 15:32:01 <07I​gorPec> this is kind of mini release 15:32:03 hmm, I thought build train would build all kernels 15:32:06 <07I​gorPec> where most of the things remain the same 15:32:31 <07I​gorPec> it will build all of them, but i don't need to use them all 15:32:50 you mean, it would build all, but you would only publish to repo a few of them? 15:32:52 <07I​gorPec> anyway this part is semi manual 15:33:02 <07I​gorPec> yes 15:33:05 <07I​gorPec> that 15:33:09 ahn ok took me a while, sorry 15:33:21 <07I​gorPec> this is some sane middle way to me 15:33:34 <07I​gorPec> a compromise 15:34:09 yeah. I'd say fire off the train, lets see what builds and what not. Also, what builds cleanly (all patches applying green/ok) 15:34:23 <07I​gorPec> everything builds, that's not a question 15:34:32 ahn. interesting. 15:34:48 we'd need at least to fix/hardcoded the tag:xxx for every family in the release branch 15:35:04 although sunxi's are already that way in master it looks like, others aren't 15:36:00 also, might be a good rule of thumb for "minor releases" to NOT change any current kernel, only edge (except for bugfixes on current maybe) 15:36:16 <07I​gorPec> yes, that is a lot more predicatble 15:36:42 also that makes it easier for you: build train, then rm *current*, then publish 😉 15:36:43 <07I​gorPec> something we can release with smoke tests only 15:37:35 <07I​gorPec> that could be doable 15:37:45 am I correct in assuming that minor releases allow you to change everything except kernel/uboot patches? 15:38:00 <07I​gorPec> probably a great solution. we provide everything others do, but keep stability at least the same 15:38:22 or maybe cat targets.conf | grep -v current 15:38:26 <07I​gorPec> yes 15:38:28 how and where to save changes for the kernel/uboot. are we not just going to wait a quarter for a minor release to make changes? 15:39:58 <07I​gorPec> we will probably making a pro version with "update at anytime" support 15:40:34 <07I​gorPec> this brings up too many problems 15:41:57 yep. for pro version, we need some serious repository management abilities, maybe even a full repo per-version, not only main/beta 15:42:11 eg: who knows the codename for 22.05? (I don't.) 15:42:14 <07I​gorPec> regarding u-boot and updating, u-boot packages are still per branch so this is not a problem 15:42:31 <07I​gorPec> if we skip CURRENT, others will still be update 15:42:59 <07I​gorPec> 22.08 Yapok 15:43:16 yeah. that sounds like a good compromise for now. I'll try: for 22.08, we're doing a minor release of edge stuff only. current (and legacy etc) will not be built or updated in this release 15:43:38 <07I​gorPec> legacy actually can 15:44:38 yeah, can, but I'd stick to edge to enforce the mainline / community aspect of it 15:44:50 <07I​gorPec> ok, fine for me 15:45:27 <07I​gorPec> then we don't need testers for this one 😉 Or lets say this is optional 15:45:41 <07I​gorPec> as EDGE is anyway ... EDGE 15:47:02 <07I​gorPec> and the same goes for 2nd next, 23.02 release 15:47:03 well I'll test meson64 odroid stuff ofc I also have rockpro64, quartz64a, rpi4b available out of production to test 15:47:31 and the damned tinkerboard-2 😉 15:47:52 To cut a long story short (and come to an end of the half hour meeting ^^) there will be a release with minor stuff, correct? 15:48:00 <07I​gorPec> yes, 15:48:04 Yeah we are running in circles. 15:48:11 the checklist here https://www.armbian.com/rc-testing/ requires to be filled anyway for this mini-release ? I think it is a good idea to do that anyway... 15:48:12 ^ RC Testing – Armbian 15:48:19 #agreed Release yes. Minor changes only 15:48:27 Yes fill out checklist. 15:48:29 <07I​gorPec> EDGE kernel upgrade only 15:48:49 So anything left that should be included in the meeting protocol? 15:48:51 <07I​gorPec> but full images rebuild from current build branch 15:49:19 <07I​gorPec> that second next release will be the same, so two minor, two major releases per year 15:49:42 #agreed that second next release will be the same, so two minor, two major releases per year 15:50:44 <07I​gorPec> as far as release meeting topic, we are done. a bit over schedule as always 😉 15:50:51 yup 15:50:53 <[TheBug]> #agreed Fill out rc-testing checklist as part of mini-release 15:50:56 guys, can i skip filling in https://www.armbian.com/rc-testing/? jethome has its own path with test/use release. 15:50:57 ^ RC Testing – Armbian 15:51:55 Aight 15:51:57 #endmeeting