SouthEast Linux Fest 2023 by the numbers

DNS

I am using this to analyze the logs https://github.com/Matty9191/bind-query-log-statistics
Top domain names(note I have removed some internal hostnames to only show public hostname) The numbers next to hostname is number of dns requests found. Please note that the logs was only part of Saturday and all day Sunday. The logs rolled over faster than I thought they would. There was a few internal host names that was from other peoples networks that over 10k requests in 2 days covered in the logs.

studio.youtube.com : 75924
youtube-ui.l.google.com : 61760
www.google.com : 17262
ssl.gstatic.com : 6835
signaler-pa.clients6.google.com : 6768
beacons.gcp.gvt2.com : 5725
gateway.icloud.com : 4997
connectivitycheck.gstatic.com : 4948
play.google.com : 4879
safebrowsing.googleapis.com : 4584
ping.archlinux.org : 4369
www.youtube.com : 3997
ipv4only.arpa : 3693
www.gstatic.com : 3299
detectportal.firefox.com : 3282
accounts.google.com : 3228
fonts.gstatic.com : 3136
clients4.google.com : 3066
wifiprotectcheck.mcafee.com : 3053
somafm.com : 2920
optimizationguide-pa.googleapis.com : 2815
www.googleapis.com : 2619
example.org : 2347
play.googleapis.com : 2271
googleads.g.doubleclick.net : 2135
api.steampowered.com : 2131
fedoraproject.org : 2102
mail.google.com : 2093
baideal.buadh-brath.com : 2063
location.services.mozilla.com : 2012
i.ytimg.com : 2005
connectivitycheck.grapheneos.network : 1981
gs-loc.apple.com : 1972
clients6.google.com : 1911
conncheck.opensuse.org : 1872
content-autofill.googleapis.com : 1862
graph.facebook.com : 1822
clubhouse.pubnubapi.com : 1795
adservice.google.com : 1764
_dns.resolver.arpa : 1760
www.google-analytics.com : 1737
raw.githubusercontent.com : 1719
encrypted-tbn0.gstatic.com : 1709
incoming.telemetry.mozilla.org : 1704
bag.itunes.apple.com : 1699
fonts.googleapis.com : 1690
chat.signal.org : 1667
dns.google : 1661
pagead2.googlesyndication.com : 1635
push.services.mozilla.com : 1608
youtubei.googleapis.com : 1595
ocsp.digicert.com : 1594
chat.google.com : 1568
google.com : 1558
lh3.googleusercontent.com : 1524
connectivity-check.ubuntu.com : 1508
ping.manjaro.org : 1498
grafana.com : 1482
gateway.fe.apple-dns.net : 1481
ha.thejohnweb.com : 1471
discovery-v6.syncthing.net : 1469
ocsp.pki.goog : 1467
beacons.gvt2.com : 1394
epdg.epc.mnc260.mcc310.pub.3gppnetwork.org : 1378
arcus-uswest.amazon.com : 1369
contile.services.mozilla.com : 1359
www.facebook.com : 1352
ib.adnxs.com : 1339
app-measurement.com : 1337
trace.svc.ui.com : 1334
mtalk.google.com : 1319
outlook.office365.com : 1292
gitlab.corp.redhat.com : 1270
addons-pa.clients6.google.com : 1267
fw.bme1.me : 1255
v10.events.data.microsoft.com : 1241
jnn-pa.googleapis.com : 1197

Queries per hour:
00: 19052
01: 10107
02: 10661
03: 11774
04: 10447
05: 8977
06:10114
07: 10805
08: 22667
09: 34093
10: 32292
11: 45774
12: 31282
13: 91411
14: 110778
15: 69533
16: 44638
17: 40701
18: 40361
19: 42923
20:
37136
21: 46039
22: 33685
23: 24407

Wifi

Below is total known clients for each category that connected to the one of the SELF SSIDs.
6ghz (note this year we only had 1 ap that supported this band) = 2
wifi3(802.11g) = 0
wifi4(802.11n) = 37
wifi5(802.11ac) = 178
wifi6(802.11ax) = 44
1×1 = 65
2×2 = 192
3×3 = 3
4×4 = 0
Total client 260

WAN


ATnT(1000m down, 1000m up)
peak down = 721
peak up = 148
95% rate down 194
95% rate up 38
Total down 1.27tb
Total up 420gb

Lancache

Cache added during the event 409.3
cache served during the event 572.5
Hit/miss ratio 71% miss 28% hit
total clients 136
Amount served to top client 151gb
Total downloaded for Steam 470gb
Total downloaded for Wsus 6.3gb
Total downloaded for linux mirrors 21.7
Top 3 games downloaded in steam by total gb
1. Shetland Pony
2. TF2
3. Quarters

Wired traffic

Below is total up and down
Total traffic from firewall to core sw 6.41TB
Total traffic from core sw to balllroomA/C 1.26TB
Total traffic from core sw to balllroomD 1.75Tb
Total traffic from core sw to balllroomJR 617GB
Total traffic on wired public vlan 1.22TB
Total traffic on WPA SSID vlan 534GB
Total traffic on open SSID vlan 198GB

DWCC stats from SouthEast Linux Fest

I deployed DWCC(https://github.com/zunder1990/Distributed-Wifi-Capability-Collector) for the first time at a large event. I was only able to get one node running, and was only able to capture packets on the following channels: 1,6,11,36,52,56,100,116. With one node running for two days and only capturing the management and control part of the wireless packets, I got 17gb of PCAPs.

Here are the results (only counting when client was talking with AP and not when client was probing broadcast):

2.4ghz clients204
5ghz clients90Note: I was only able capture a few of the 5ghz channels
802.11k clients110 Provides information to discover the best available access point
BSS Transition/802.11r/FT231Allows for faster roaming between APs
QOS map53RFC 8325 tells you if the client is able to set DSCP value on packets it sends
receive frames from mu-mimo AP31Says if client is able to receive packets send from AP while doing mu-mimo
receive frames from single user beamforming AP51Says if the client is able to understand an AP trying to beamform
Transmission of STBC-coded frames28Space-Time Block Coding (STBC) can be used when the number of radio chains exceeds the number of spatial streams.
Extended Channel Switching80Provides a mechanism for an access point to notify the stations connected to it of its intention to change channels or to change channel bandwidth.
Orthogonal Frequency-Division Multiplexing OFDM126
WNM-Sleep Mode45Power save mode
Multiple BSSID13Client can understand when one than one BSSID is in a beacon packet from an AP
802.11w40 Protected Management Frames 

interworking/802.11u
43Related to hotspot2.0

wnm notification
59Related to 802.11v

receive LDPC-encoded frames
51Clients able to understand frame encoded in low-density parity-check, related to Transmission of STBC-coded frames

Timing Measurement
9802.11mc, allows the client to tell how far it is from the AP,  3.3 nanoseconds per meter

SSID List
45

Vote boxes

After a trip to Ikea they had simple voting system near the exits of the store. That got us to think so we made a box that we would place at the exit of session to allow guests to rate the session. We simply ask if the user “loved it” , “was neutral about it” or “hated it”

For the hardware we use esp8266 nodemcu running esphome much like the speaker lights. The vote boxes connects to the staff wifi and will report the votes using mqtt to node red. We count a love it as +1, neutral as 0 and hated as -1 to give us a final vote.

esphome:
  name: ballroomavote
  platform: ESP8266
  board: nodemcuv2

wifi:
  ssid: 'x'
  password: 'x'
  manual_ip:
   static_ip: x.x.x.x
   gateway: x.x.x.x
   subnet: 255.255.255.0
# Enable logging
logger:
ota:
  password: 'x'
sensor:
  - platform: wifi_signal
    name: "balllroomAvote WiFi Signal Sensor"
    update_interval: 60s
switch:
  - platform: gpio
    pin: D2
    name: "vote LED"
    id: voteled

binary_sensor:
  - platform: gpio
    pin:
      number: D3
      mode: INPUT_PULLUP
    name: "pos  Button"
    filters:
      - delayed_on: 10ms
    on_press:
      then:
        - switch.toggle: voteled
        - delay: 2ms
        - switch.toggle: voteled
        - mqtt.publish:
             topic: vote/A
             payload: "pos"


  - platform: gpio
    pin:
      number: D6
      mode: INPUT_PULLUP
    name: "natural  Button"
    filters:
      - delayed_on: 10ms
    on_press:
      then:
        - switch.toggle: voteled
        - delay: 2ms
        - switch.toggle: voteled
        - mqtt.publish:
             topic: vote/A
             payload: "natural"


  - platform: gpio
    pin:
      number: D5
      mode: INPUT_PULLUP
    name: "neg  Button"
    filters:
      - delayed_on: 10ms
    on_press:
      then:
        - switch.toggle: voteled
        - delay: 2ms
        - switch.toggle: voteled
        - mqtt.publish:
             topic: vote/A
             payload: "neg"

mqtt:
  broker: x.x.x.x
  keepalive: 15s
  username: x.x.x.x
  password: x.x.x.x

Speaker Lights

We use the speaker lights to give speaker a heads up about what the automation is doing with the recording.

For the hardware we used an esp8266 nodemcu board connected to the LED traffic light style board. Here is an example of the type of LED board that we used:
https://www.amazon.com/Oak-Pine-Traffic-Display-Creative-Arduino/dp/B07R4YR2QJ

For the software we are using ESPhome and speaker lights assembly connect to the staff wifi. Over the wifi they will connect to the MQTT server to listen for commands. We powered the speaker light using USB.

Here is what the lights do:

  • 1 min before the start time of the session the red LED will flash
  • At start time the red LED will stop flashing and turn the green LED on. With the Green LED meaning that the recording has started.
  • 5 mins before the end of the session the yellow LED will start to flash
  • At the end time of the session the green LED will turn off. The yellow LED will turn off and the red LED will start to flash
  • 5 min after the end of the session the red LED will stop flashing and turn to solid

Below is what it looks like before it was placed into a case.

Here is the esphome config file that we use.

esphome:
  name: ballrooma_speaker_light
  platform: ESP8266
  board: nodemcuv2

wifi:
  ssid: 'xxx'
  password: 'x'
  manual_ip:
   static_ip: x.x.x.x
   gateway: x.x.x.x
   subnet: 255.255.255.0
# Enable logging
logger:
ota:
  password: 'x'
switch:
  - platform: gpio
    pin: D2
    name: "Red LED"
    state_topic: light/a/
    command_topic: light/a/red
  - platform: gpio
    pin: D2
    name: "Red Flash LED"

    id: Red_pin
  - platform: template
    name: "Red Flash LED"
    optimistic: yes
    id: redtemp
    state_topic: light/a/
    command_topic: light/a/redflash
    turn_on_action:
    - while:
       condition:
        lambda: 'return true;'
       then:
       - switch.turn_on: Red_pin
       - delay: 500ms 
       - switch.turn_off: Red_pin
       - delay: 500ms
    turn_off_action:
    - switch.turn_off: Red_pin



  - platform: gpio
    pin: D1
    name: "Yellow LED"

    id: yellow_pin
  - platform: template
    name: "Yellow LED"
    optimistic: yes
    id: yellowtemp
    state_topic: light/a/
    command_topic: light/a/yellow
    turn_on_action:
    - while:
       condition:
        lambda: 'return true;'
       then:
       - switch.turn_on: yellow_pin
       - delay: 500ms 
       - switch.turn_off: yellow_pin
       - delay: 500ms
    turn_off_action:
    - switch.turn_off: yellow_pin


  - platform: gpio
    pin: D3
    name: "Green LED"
    state_topic: light/a/
    command_topic: light/a/green
    
mqtt:
  broker: x.x.x.x
  keepalive: 15s
  username: x
  password: x

Networking for SELF

In case you missed it, I have done an overview of everything we do for SELF here

For the SouthEast Linux Fest we take over the event space of the Sheraton Charlotte Airport Hotel to host our 3 day event. I have been helping with SELF for the past 5-6 years and in those years we would get to the hotel on the Thursday morning before the event and have all day to setup the network. We go live with the network Friday 8 am as that is the start of the event. With the permission of the hotel staff we turn off and power down the hotel’s wifi system in the event spaces were we will be setting up our own wifi. Doing this helps with reducing RF interference.

The logistics of getting there.

I live about a hour away from the hotel that we host SELF at as a result I really do not want to make two trips each way. I am fortunate to have a Yukon and large trailer so I am able to load everything in one trip. I have found those large Husky tool chests very nice. This was gearing up to SELF 2019.

In 2019 I took 9 displays

Internet connection

Over the years we had a few different forms of internet connection all around a coax connection from the local cable company. When I first started to help we started with 100×5 internet connection. Next we did 3 bonded 100×5 connections to gain more upload speed for videos. We progressed to what we currently use, a 940×35 coax connection. I wish we could have fiber but can’t afford it. We are trying to get a local Wireless ISP to get us a connection. If that happens then we should be able to get a 500×500 connection. We also run a full dual stack ipv4/ipv6 network.

Layer 1 the cabling

Over the years we run own network cabling both cat6 and fiber. We are able to leave the cabling in the ceiling year to year allowing to reuse and built up over time the cabling that we have in place. In 2019 we expand into a new part of the hotel so we needed to extend the network to new wing. This wing was connected to the our current wing with a 5 foot in diameter underground pipe that carried services between the two different parts of the hotel. In this tunnel we were able to run a 800 feet of fiber which allowed us to use 10gbit networking between our switches.

We deploy temporary cabling to run between the end devices and the switches. Those temporary cable runs are mostly made of cable trunks where we will group 2-8 ethernet cables together as a single run. We did this because we found that we were running multiple cables to the same location. Right now we have 67 cable runs which total over 7,000 feet of ethernet. Below you can see how we labeled each trunk cable. The 24 the cable ID and the 64ft is how long it is. All of the cable’s info is saved in a spreadsheet with the ID number as the reference point. Each port is then labeled started with A. So you get something like port 24A. Both ends of the cables are labeled like this.

Switching

Right now we use 5 Dell Powerconnect 5524 POE switches that we got from ebay for between $100-150 each. We use the 10gbit ports as trunk ports between switches. Currently the switches are setup by hand using the built in GUI and CLI, we are hopping to change that soon. Also in the future we are looking at some Brocade switch to get more 10gbit ports and some 40gbit ports.

Routing

At this time we only do inner-vlan routing on our PFsense firewall. The PFsense box has 4x 1gbit interfaces that get lagged together and linked to the switches. When we get more 10gbit ports we will be moving to 10gbit LAN link. The PFsense also has another 1gbit interface that we use as our uplink to the coax modem.

WIFI

Right now we use 20 Unifi APs bulk of them the AC-pro model but we also use nano-HD, AC-HD, AC-lite, AC mesh. All APs are powered from the POE switches and backhauled using ethernet. We host the Unifi controller onsite. We run 3 SSID at SELF, staff, public fast(5ghz only) and public slow(2.4ghz only). We use social engineering with the names, to encourage users to connect to the 5ghz only network.

Servers

We run all services that are needed for the event onsite.

namecpuramOSModelroles
Bigserver2x E5620 (8 core total)96gbUbuntu 18.04dns, nfs storage, mqtt, mysql
gameserver2x L5640 (12 cores total)24gb ramUbuntu 16.04lan cache, game servers
desktopw353016gbWindows 10camlytics
noderedi5-2400S4gbUbuntu 18.04node red, unifi-poller
NVRcore i7 4770s8gbWindows 10blue-iris

Lan party area

We host a lan party area at the Linux fest. In this area we offer tables with power and wired network drops. We host a few game servers onsite for games that support Linux well. Since the guest wireless and wired lan party network are open guests are allowed to run there own game servers if they wish. We built a power monitoring rig to allow us to track power usage to make sure that we are not about trip a breaker. From the hotel’s connection we get 2x 20amp breakers, we take one lead to each breaker. Inside of the gray box we have Shelly EM and two CT clamp. The CT clamp will give us, in real time, the number of amps running to each breaker. Each breaker will connect to one of the quad set of outlets.

Note that we did put outlet covers on after this picture was taken

NOC setup

For our NOC setup we have a shared space in the admin/command room. This is where we place the server and have all of the cables run back to. In the picture you can see the cabling coming down from the ceiling. For SELF 2021 we will be moving the server into a closet to reduce the noise and heat of the NOC area.

Intro to the network and automation of the SouthEast Linux Fest

This is to serve as introduction to the network and automation of the SouthEast Linux Fest and documentation of said setup. This will link to more details posts of each component. This is written as of October 2020 and will be updated over time.

The SouthEast Linux Fest (SELF) is an 3 day event that is held in Charlotte, NC each summer and pull together around 800 attendees and speakers. We have over 70 sessions in 5 tracks talking all about Linux and open source. SELF started in 2009 and held in person each year with the exception of 2020 due to Covid-19. We are hopping that we can back in person for Summer 2021. My role with SELF is to lead the team that handles all onsite tech needs around the network, servers, wifi, and video recording.

Here is a list and overview of the services that we offer for the event

Recording the talks
We record the speaking session will try to have the sessions uploaded to Youtube with in one hour of the end of the session.
Here is example video from a session that I gave in 2019  https://youtu.be/eFNsEpwpRek 

For SELF 2021 we will be uploading not only to Youtube but to the Internet Archive and Peertube as well.


OBS setup
Being an Linux/open source event we try to use open source where ever we can. We use Open Broadcast Studio for all of the video recording. For SELF 2021 we will partnering with the folks at http://www.asknoahshow.com/ live stream each sessions allow with uploading the video post session. 

Speaker Lights
Since the recording of sessions is all automatic we needed a way to let the speaker know about the timing info. We built 3 led stop light style indictors that give the speaker a heads up.

More details about it here


People counting
We placed a camera over each doorway leading to a session room, this camera would be pointing straight down as people walked into each room. We connect the cameras to some software that count the people entering the session.

Vote boxes
We placed a box at the exit of each session room with 3 buttons: loved the talk, neutral or hated the talk.

More details about it here


Displays
At the hotel were we are hosted, they have built in TV displays outside of each room that are used for digital signage. SELF 2021 we will be repurposing the displays for our use.

 
Network
We bring in our own internet connection to the hotel and use none of the hotel’s house wiring. We roll out 2k feet of fiber and 7k feet of copper to support the network. We install 8 servers and 20 APs to give wifi services to our lan party area, staff, vendors and all guests. 

More details about the networking here

Monitoring

We monitor all facets of the network, wifi, and automation. We also have a public display where we will show some of the stats, such as how many wifi users, the total number of votes logged for the presentations, how many attendees have entered presentations, etc.

Automation
The heart of our automation system is Node-Red, we use it to automate all of the above with the exception of the network. I hope to change that one day.

About Zach Underwood

I have worked in both a commercial data centers, local government and ISPs. I have experience includes setup of VMware stack using ESX and Vcenter to reduce the number of servers, setup internal WI-FI systems, designed GPO for windows desktop, and deployed Linux thin clients for public kiosks. I have also provided server administration and support, desktop user support, and computer equipment troubleshooting.

  • Red Hat Certified System Administrator Ver. 7 License number:100-206-556 January 2015
  • Red Hat Certified Engineer Ver.6 License number:100-206-556 June 2011
  • Red Hat Certified System Administrator Ver. 6 License number:100-206-556 July 2011
  • Red Hat Certified System Administrator Ver. 5 License number 100-206-556 August 2010
  • Red Hat Certified Technician Ver. 5 License number:100-206-556 August 2010
  • Ubiquiti airMAX-Certified Admin September 2013
  • Commtrain/NATE Certified tower climber March 2014

Videos that I have done

Greenville Tech ad that aired on TV

Linux user group talk about BGP and Net Neutrality

Interview I did with Jupiter Bradcasting about Wireless ISP and tower climbing

This is a talk that I gave at the South East Linux Fest 2015

This is a talk that I gave at the South East Linux Fest 2015

This is a talk that I gave at the South East Linux Fest 2014

Interview I did with Jupiter Bradcasting