Using SiLK (Peak Idle Bandwidth for Storage example)

I needed to determine the average traffic seen at the internet gateway in order to scope full packet capture, Bro and flow data.
Then i needed to find traffic to eliminate and gain storage retention.

Mon-Fri are peak, and Sat-Sun are idle times.

Export flows for a 24 hour period to a binary file
rwfilter –start-date=2014/10/X –proto=0- –type=all –pass=peakmonday.raw

Find bandwidth total for that day in bytes
cat peakmonday.raw | rwcount –bin-size=86400

To get Gigabytes per day, divide results by 1024, x to the 3rd
(divide by 1024 3 times, kilobytes, megabytes, gigabytes)

Do this for Mon-Fri in one set and Sat-Sun in another. (Peak/Idle)
Multiply the highest peak result by 5.
Multiply the highest idle result by 2.
Add the two results together.
Find the average daily total by dividing that number by 7.

You can divide this again by 24 to find the average amount of PCAP GB generated per hour.

Ran this command to find where the sensors reside.
rwsiteinfo –fields=sensor,describe-sensor
Sensor|                                                              Sensor-Description|
 SENSOR1|Circuit 1, Active, Backup n/a|
 SENSOR2|Circuit 2, Active, Backup n/a|

Ran commands for Thursday since it the busiest day.

PEAK – Thursday 2014-09-25
rwfilter –sensors=SENSOR1,SENSOR2 –start-date=2014/09/25 –proto=0- –type=all –pass=peakthurs.raw &
cat peakthurs.raw | rwcount –bin-size=86400 > peakthurs.txt
11,100 GB per day. 
55,500 GB per week

IDLE – Saturday 2014-09-27
rwfilter –sensors=SENSOR1,SENSOR2 –start-date=2014/09/27 –proto=0- –type=all –pass=idlesat.raw &
cat idlesat.raw | rwcount –bin-size=86400 > idlesat.txt
               Date|        Records|               Bytes|          Packets|
2014/09/27T00:00:00|   182673680.01|    3227205730820.63|    5009734916.20|
3005 GB on Saturday 2014-09-27. 
6010 GB per weekend.

55500 + 6010 = 61510 / 7 

8787 GB per day average at the gateway.

Next we can start to filter out what we don’t need.
Find the top protocols by source and destination during peak weekdays.
cat peakthurs.raw | rwstats –fields=sport –top –count=8 –value=bytes > peakthurssportProto.txt
INPUT: 400591595 Records for 65536 Bins and 11947859070439 Total Bytes
OUTPUT: Top 8 Bins by Bytes
sPort|               Bytes|    %Bytes|   cumul_%|
   80|       4523070386113| 37.856744| 37.856744|
  443|       2931570222744| 24.536364| 62.393108|
 4500|       1104216249438|  9.241959| 71.635067|

cat peakthurs.raw | rwstats –fields=dport –top –count=8 –value=bytes > peakthursdportProto.txt
INPUT: 400591595 Records for 65527 Bins and 11947859070439 Total Bytes
OUTPUT: Top 8 Bins by Bytes
dPort|               Bytes|    %Bytes|   cumul_%|
    0|        935831821423|  7.832632|  7.832632|
  443|        595528106011|  4.984392| 12.817024|
 4500|        448105483380|  3.750509| 16.567532|

Find the top protocols by source and dest during idle weekend.
cat idlesat.raw | rwstats –fields=sport –top –count=8 –value=bytes > idlesunsportProto.txt
INPUT: 182682920 Records for 65536 Bins and 3244533760545 Total Bytes
OUTPUT: Top 8 Bins by Bytes
sPort|               Bytes|    %Bytes|   cumul_%|
   80|       1231602802154| 37.959315| 37.959315|
  443|        571667861517| 17.619415| 55.578730|
    0|        377344171218| 11.630151| 67.208881|

cat idlesat.raw | rwstats –fields=dport –top –count=8 –value=bytes > idlesundportProto.txt
INPUT: 182682920 Records for 65531 Bins and 3244533760545 Total Bytes
OUTPUT: Top 8 Bins by Bytes
dPort|               Bytes|    %Bytes|   cumul_%|
    0|        376346699708| 11.599408| 11.599408|
  443|        290123245079|  8.941909| 20.541316|
 4500|        133027626116|  4.100054| 24.641370|

We can see that port 0(ICMP), 80(HTTP), 443(SSL), and 4500(IPSEC over NAT-T) consume most of our traffic.  We cannot see encrypted traffic so let’s eliminate port 443, and 4500 to see what our sizes would be.

cat peakthurs.raw | rwfilter –input-pipe=stdin –aport=443,4500 –fail=stdout | rwcount –bin-size=86400 > peakthursnoencrypt.txt
               Date|        Records|               Bytes|          Packets|
2014/09/25T00:00:00|   307658485.07|    6921598440639.58|    9131918500.41|
6446 GB per day.
32230 GB per weekdays

cat idlesat.raw | rwfilter –input-pipe=stdin –aport=443,4500 –fail=stdout | rwcount –bin-size=86400 > idlesatnoencrypt.txt
               Date|        Records|               Bytes|          Packets|
2014/09/27T00:00:00|   156578704.52|    2000601612161.50|    2803744983.17|
1863 GB per weekend day
3726 GB per weekend

32230 + 3726 = 35956 / 7

5136 GB per day at the gateway.

Cutting out encrypted traffic give us around 40% more storage space.

Based on full PCAP being 100%, Bro data being 10% and SiLK data being 1%.

Full PCAP requires 5 TB per day
Bro data requires 500 GB per day
SiLK Session data requires 50 GB per day.