How Do AI Workloads Use the Network?

The GPUs arrived, but the data hasn't

When a team plans AI infrastructure, the top line of the budget is always GPUs. How many to buy and which model to buy takes weeks of discussion. The network usually comes last, with a quick "the internet will work, right?"

The problem shows up after the hardware is installed. The training data lives somewhere else, and moving it takes days. The GPUs sit powered on and waiting the whole time. Or the opposite happens: the model is finished and in production, it generates answers quickly, but those answers take a while to reach the user.

In both cases the cause is the network. And they are two different problems.

Training moves a lot at once

Training traffic comes in large chunks. You pull in a dataset, push checkpoints to storage partway through training, and ship the finished model to wherever it will be served. The line is quiet most of the time and completely full during a transfer.

What matters here is how wide the port is. Moving the same 100TB takes this long depending on port speed.

Port	Time to move 100TB
1G	About 9 days
10G	About 22 hours
100G	About 2 hours 13 minutes
400G	About 33 minutes

These are theoretical figures that assume the port is used at full capacity. In practice it is slower, depending on storage read and write speed, the transfer tool, and the circuit on the other end. The difference in order of magnitude still holds. A job that takes more than a week on 1G is done by the time you get back from lunch on 100G.

The question to ask is not "what is our average usage" but "how long can we afford to wait during our largest transfer." Time that GPUs spend idle waiting for data is a cost too.

Where does the data come from?

The path matters as much as the port size. Training data usually sits in cloud storage or in another data center. Even with a 100G port on your side, if the stretch in between is a congested route through several carriers, your speed is set by the narrowest point.

So when you evaluate a network provider, it is worth asking these questions along with the port speed.

Is there a direct connection to where my data lives (the cloud, another data center)?
If not, how many networks does the traffic pass through on the way?
If one path is blocked, is there another route?

If the answer is "direct connection," transfer times become predictable. Over several hops of public internet, the same job takes two hours one day and six hours another.

Inference has to answer quickly every time

Once a model goes into production, the traffic pattern flips. Each request is small. But requests arrive constantly, and each one has to come back fast.

The response time a user experiences is the time the model takes to produce an answer plus the time spent traveling there and back. However fast you make the model, if the round trip takes 100ms, that part cannot be reduced. In services where even a slightly late response feels awkward, such as voice conversation or real-time translation, the difference is plainly visible.

What you look at here is distance, not port size. More precisely, how many networks sit between the user's carrier and your server. If most of your users are in Korea, a circuit that peers directly with Korean carriers and IXs has the advantage. A path that detours overseas and comes back has high latency no matter how fast the link is.

Handling both on one circuit

If training and inference run in the same place, the circuit has to handle both at the same time.

	Training	Inference
Traffic shape	Occasional, very large	Constant, small
What matters	Port size, path to the data	Latency to the user
When it goes wrong	GPUs wait for data	Users wait for answers

One thing to watch for: while a large transfer fills the circuit, inference responses can slow down. It is safer to separate the two in advance, by leaving headroom on the port, moving large transfers to hours when few users are online, or capping the transfer rate.

Look at the billing model too

For traffic that is occasional but large, the cost varies a lot with the billing model.

95th percentile billing, which is common for network circuits, measures usage every 5 minutes over a month, discards the top 5% of samples, and bills on the next highest value. In a 30-day month, 5% is 36 hours. That means heavy transfers that add up to less than 36 hours a month do not show up in the bill.

If you move training data a few times a month for a few hours each time, 95th percentile billing is a good fit. If you fill the port for several hours every day, a flat rate or a commit may work out better. Measure which kind of workload you have first, then choose.

What to ask when choosing a circuit

To sum up, you choose a circuit for AI infrastructure on these four points.

How large is your biggest transfer, and how long can you wait for it? (port size)
Does the circuit reach where that data lives directly? (path)
How long does it take from your users' carriers to your server? (latency)
How many hours a month do you use the circuit heavily? (billing model)

With LETO IP Transit you choose a 1G, 10G or 100G port, and if 100G is not enough we set up a 400G port. It peers directly with the major Korean carriers and IXs, so latency within Korea is in the single-digit ms range, and you pick the billing model that matches your traffic shape: commit, 95th percentile billing, or flat rate.

Explore IP Transit →