How to implement a 'Load Balancing' like service in Single Windows 2003 Server?

Discussion:

(too old to reply)

a***@gmail.com

2006-10-11 00:00:23 UTC

Hi there,

I got a new project to implement a 'Load Balancing' like product on
single Windows 2003 Server. Basically, here is the way the whole system
works:

1. We have a web server which need to serve massive clients via HTTP
protocol;
2. We want to run this web server as multiple instance process/exe and
each of them is listening to different port on the same machine, like
8001, 8002, 8003...etc;
3. This 'Load-Balancing' service will be running on the same machine as
these web server instances and listen to port 80. Therefore it's the
only one who gets the request from clients and dispatch the request to
one of the web servers who is relatively free base on certain algorithm
and then pass its response to client during the same session;
4. So everything is running *independantly* on the same machine but we
can take advantage of multi-core processor and do some load balancing
work;

Since I dont have any experience on implementing this kind of service.
Could anybody kindly give me some hints?

I'd specially appreciate Information like

1. What kind of Win32 technology do I need to know to be able to do
this stuff?
2. How should the whole system work in term of socket / thread / I/O?
3. Are there any existing products / source code / web sites which can
do the same/similar thing?

Thanks for your time!

Arkady Frenkel

2006-10-11 10:54:35 UTC

Permalink

W2003 Server opposite to client Oses already have
Network Load Balancing (NLB) driver inside, you can check info about it on
MSDN e.g.

Arkady

Post by a***@gmail.com
Hi there,
I got a new project to implement a 'Load Balancing' like product on
single Windows 2003 Server. Basically, here is the way the whole system
1. We have a web server which need to serve massive clients via HTTP
protocol;
2. We want to run this web server as multiple instance process/exe and
each of them is listening to different port on the same machine, like
8001, 8002, 8003...etc;
3. This 'Load-Balancing' service will be running on the same machine as
these web server instances and listen to port 80. Therefore it's the
only one who gets the request from clients and dispatch the request to
one of the web servers who is relatively free base on certain algorithm
and then pass its response to client during the same session;
4. So everything is running *independantly* on the same machine but we
can take advantage of multi-core processor and do some load balancing
work;
Since I dont have any experience on implementing this kind of service.
Could anybody kindly give me some hints?
I'd specially appreciate Information like
1. What kind of Win32 technology do I need to know to be able to do
this stuff?
2. How should the whole system work in term of socket / thread / I/O?
3. Are there any existing products / source code / web sites which can
do the same/similar thing?
Thanks for your time!

Alexander Nickolov

2006-10-11 16:53:03 UTC

Permalink

And what is this supposed to achieve? Your web server likely
already can utilize all your CPUs anyway. Spawning multiple
processes and introducing a proxy will only serve to decrease
your web server's performance (rather significantly I might add).

Now if you want to distribute the load on multiple servers, that's
another story. You don't need to write anything as whatever you
write will be the new bottleneck. Check out our Citrix Netscaler
product line. It's perfectly suited for that problem:

http://www.citrix.com/netscaler
--
=====================================
Alexander Nickolov
Microsoft MVP [VC], MCSD
email: ***@mvps.org
MVP VC FAQ: http://www.mvps.org/vcfaq
=====================================

a***@gmail.com

2006-10-11 20:44:43 UTC

Permalink

Hi Alex, thanks for your info. There is a fact that our web server is
single-threaded with the ability to server multiple connections and
also there is a core component in the web server used to process
client's request is single-threaded too. Therefore to fully utilize the
current powerful CPU specially those latest multi-core CPUs, we came
out this design. However, we will still make our system scalable to
able to expand to multiple servers based on customer's requirements and
the real system scale.

I'd appreciate if you can point out some hints on the implemetation of
this kind of product. Thanks!

Post by Alexander Nickolov
And what is this supposed to achieve? Your web server likely
already can utilize all your CPUs anyway. Spawning multiple
processes and introducing a proxy will only serve to decrease
your web server's performance (rather significantly I might add).
Now if you want to distribute the load on multiple servers, that's
another story. You don't need to write anything as whatever you
write will be the new bottleneck. Check out our Citrix Netscaler
http://www.citrix.com/netscaler
--
=====================================
Alexander Nickolov
Microsoft MVP [VC], MCSD
MVP VC FAQ: http://www.mvps.org/vcfaq
=====================================

Alexander Nickolov

2006-10-12 00:39:04 UTC

Permalink

Ah, so it's not an off-the-shelf web server, but something you
wrote, correct? In that case, may I suggest that the effort to
write the proxy may be better spent redesigning your server
to be multi-threaded? You'll likely want to change your socket
usage model as well. I'd guess you use async sockets (that's
the only somewhat viable model for a single-threaded server
as the other models are limited to 64 sockets). This doesn't
scale well due to the nature of the message loop.

At any rate, your original idea is a fairly straightforward
proxy server. You need to implement it efficiently of course,
so learn about overlapped socket I/O and I/O completion
ports (e.g. using a thread pool for overlapped I/O).

And just to be exhaustive, my original Netscaler proposal
would work for you too, though I admit it's probably an
overkill at this stage...
--
=====================================
Alexander Nickolov
Microsoft MVP [VC], MCSD
email: ***@mvps.org
MVP VC FAQ: http://www.mvps.org/vcfaq
=====================================

Post by a***@gmail.com
Hi Alex, thanks for your info. There is a fact that our web server is
single-threaded with the ability to server multiple connections and
also there is a core component in the web server used to process
client's request is single-threaded too. Therefore to fully utilize the
current powerful CPU specially those latest multi-core CPUs, we came
out this design. However, we will still make our system scalable to
able to expand to multiple servers based on customer's requirements and
the real system scale.
I'd appreciate if you can point out some hints on the implemetation of
this kind of product. Thanks!

William Stacey [C# MVP]

2006-10-12 01:34:40 UTC

Permalink

I agree Alexander. OP - Forcing a proxy ontop single threaded processes
(e.g. unix like), just to handle multiple threads seems to be a strange
design and a lot of unneeded overhead. It seems to me that if you want to
keep a single threaded main server loop, then it seems a
Half-Sync/Half-Async pattern would be better. So all network IO uses async
pattern. Your reads do async reads for "whole" messages (i.e. more async
reads till whole message), then post message to InQ. InQ is a blocking
queue (could also be bounded). Your server thread would just loop on
Dequeue of InQ popping messages, process them, and write reply using async
writes - loop again. This is a neat design as you can easily add more
worker threads if needed (i.e. a thread pool). Other worker threads just
block on InQ.Dequeue() if no work. That way you can easily use more cpus by
adding worker threads to the pool, maybe N+1 threads. You can also do other
interesting stuff like push-back with bounded queue size, inject messages
back into the stream (i.e. messages to the threads in the pool) by posting
messaging in the inQ, etc. Here is a paper showing the pattern:
http://www.cs.wustl.edu/~schmidt/PDF/PLoP-95.pdf
--
William Stacey [C# MVP]

"Alexander Nickolov" <***@mvps.org> wrote in message news:***@TK2MSFTNGP04.phx.gbl...
| Ah, so it's not an off-the-shelf web server, but something you
| wrote, correct? In that case, may I suggest that the effort to
| write the proxy may be better spent redesigning your server
| to be multi-threaded? You'll likely want to change your socket
| usage model as well. I'd guess you use async sockets (that's
| the only somewhat viable model for a single-threaded server
| as the other models are limited to 64 sockets). This doesn't
| scale well due to the nature of the message loop.
|
| At any rate, your original idea is a fairly straightforward
| proxy server. You need to implement it efficiently of course,
| so learn about overlapped socket I/O and I/O completion
| ports (e.g. using a thread pool for overlapped I/O).
|
| And just to be exhaustive, my original Netscaler proposal
| would work for you too, though I admit it's probably an
| overkill at this stage...
|
| --
| =====================================
| Alexander Nickolov
| Microsoft MVP [VC], MCSD
| email: ***@mvps.org
| MVP VC FAQ: http://www.mvps.org/vcfaq
| =====================================
|
| <***@gmail.com> wrote in message
| news:***@k70g2000cwa.googlegroups.com...
| > Hi Alex, thanks for your info. There is a fact that our web server is
| > single-threaded with the ability to server multiple connections and
| > also there is a core component in the web server used to process
| > client's request is single-threaded too. Therefore to fully utilize the
| > current powerful CPU specially those latest multi-core CPUs, we came
| > out this design. However, we will still make our system scalable to
| > able to expand to multiple servers based on customer's requirements and
| > the real system scale.
| >
| > I'd appreciate if you can point out some hints on the implemetation of
| > this kind of product. Thanks!
| >
| > Alexander Nickolov wrote:
| >> And what is this supposed to achieve? Your web server likely
| >> already can utilize all your CPUs anyway. Spawning multiple
| >> processes and introducing a proxy will only serve to decrease
| >> your web server's performance (rather significantly I might add).
| >>
| >> Now if you want to distribute the load on multiple servers, that's
| >> another story. You don't need to write anything as whatever you
| >> write will be the new bottleneck. Check out our Citrix Netscaler
| >> product line. It's perfectly suited for that problem:
| >>
| >> http://www.citrix.com/netscaler
| >>
| >> --
| >> =====================================
| >> Alexander Nickolov
| >> Microsoft MVP [VC], MCSD
| >> email: ***@mvps.org
| >> MVP VC FAQ: http://www.mvps.org/vcfaq
| >> =====================================
| >>
| >> <***@gmail.com> wrote in message
| >> news:***@m7g2000cwm.googlegroups.com...
| >> > Hi there,
| >> >
| >> > I got a new project to implement a 'Load Balancing' like product on
| >> > single Windows 2003 Server. Basically, here is the way the whole
system
| >> > works:
| >> >
| >> > 1. We have a web server which need to serve massive clients via HTTP
| >> > protocol;
| >> > 2. We want to run this web server as multiple instance process/exe
and
| >> > each of them is listening to different port on the same machine, like
| >> > 8001, 8002, 8003...etc;
| >> > 3. This 'Load-Balancing' service will be running on the same machine
as
| >> > these web server instances and listen to port 80. Therefore it's the
| >> > only one who gets the request from clients and dispatch the request
to
| >> > one of the web servers who is relatively free base on certain
algorithm
| >> > and then pass its response to client during the same session;
| >> > 4. So everything is running *independantly* on the same machine but
we
| >> > can take advantage of multi-core processor and do some load balancing
| >> > work;
| >> >
| >> > Since I dont have any experience on implementing this kind of
service.
| >> > Could anybody kindly give me some hints?
| >> >
| >> > I'd specially appreciate Information like
| >> >
| >> > 1. What kind of Win32 technology do I need to know to be able to do
| >> > this stuff?
| >> > 2. How should the whole system work in term of socket / thread / I/O?
| >> > 3. Are there any existing products / source code / web sites which
can
| >> > do the same/similar thing?
| >> >
| >> > Thanks for your time!
| >> >
| >
|
|

a***@gmail.com

2006-10-12 06:10:02 UTC

Permalink

Hi William, your idea is very cool and I'll keep it in mind in my
implementation. Thanks for your input and the link for the document!!!

Post by William Stacey [C# MVP]
I agree Alexander. OP - Forcing a proxy ontop single threaded processes
(e.g. unix like), just to handle multiple threads seems to be a strange
design and a lot of unneeded overhead. It seems to me that if you want to
keep a single threaded main server loop, then it seems a
Half-Sync/Half-Async pattern would be better. So all network IO uses async
pattern. Your reads do async reads for "whole" messages (i.e. more async
reads till whole message), then post message to InQ. InQ is a blocking
queue (could also be bounded). Your server thread would just loop on
Dequeue of InQ popping messages, process them, and write reply using async
writes - loop again. This is a neat design as you can easily add more
worker threads if needed (i.e. a thread pool). Other worker threads just
block on InQ.Dequeue() if no work. That way you can easily use more cpus by
adding worker threads to the pool, maybe N+1 threads. You can also do other
interesting stuff like push-back with bounded queue size, inject messages
back into the stream (i.e. messages to the threads in the pool) by posting
http://www.cs.wustl.edu/~schmidt/PDF/PLoP-95.pdf
--
William Stacey [C# MVP]
| Ah, so it's not an off-the-shelf web server, but something you
| wrote, correct? In that case, may I suggest that the effort to
| write the proxy may be better spent redesigning your server
| to be multi-threaded? You'll likely want to change your socket
| usage model as well. I'd guess you use async sockets (that's
| the only somewhat viable model for a single-threaded server
| as the other models are limited to 64 sockets). This doesn't
| scale well due to the nature of the message loop.
|
| At any rate, your original idea is a fairly straightforward
| proxy server. You need to implement it efficiently of course,
| so learn about overlapped socket I/O and I/O completion
| ports (e.g. using a thread pool for overlapped I/O).
|
| And just to be exhaustive, my original Netscaler proposal
| would work for you too, though I admit it's probably an
| overkill at this stage...
|
| --
| =====================================
| Alexander Nickolov
| Microsoft MVP [VC], MCSD
| MVP VC FAQ: http://www.mvps.org/vcfaq
| =====================================
|
| > Hi Alex, thanks for your info. There is a fact that our web server is
| > single-threaded with the ability to server multiple connections and
| > also there is a core component in the web server used to process
| > client's request is single-threaded too. Therefore to fully utilize the
| > current powerful CPU specially those latest multi-core CPUs, we came
| > out this design. However, we will still make our system scalable to
| > able to expand to multiple servers based on customer's requirements and
| > the real system scale.
| >
| > I'd appreciate if you can point out some hints on the implemetation of
| > this kind of product. Thanks!
| >
| >> And what is this supposed to achieve? Your web server likely
| >> already can utilize all your CPUs anyway. Spawning multiple
| >> processes and introducing a proxy will only serve to decrease
| >> your web server's performance (rather significantly I might add).
| >>
| >> Now if you want to distribute the load on multiple servers, that's
| >> another story. You don't need to write anything as whatever you
| >> write will be the new bottleneck. Check out our Citrix Netscaler
| >>
| >> http://www.citrix.com/netscaler
| >>
| >> --
| >> =====================================
| >> Alexander Nickolov
| >> Microsoft MVP [VC], MCSD
| >> MVP VC FAQ: http://www.mvps.org/vcfaq
| >> =====================================
| >>
| >> > Hi there,
| >> >
| >> > I got a new project to implement a 'Load Balancing' like product on
| >> > single Windows 2003 Server. Basically, here is the way the whole
system
| >> >
| >> > 1. We have a web server which need to serve massive clients via HTTP
| >> > protocol;
| >> > 2. We want to run this web server as multiple instance process/exe
and
| >> > each of them is listening to different port on the same machine, like
| >> > 8001, 8002, 8003...etc;
| >> > 3. This 'Load-Balancing' service will be running on the same machine
as
| >> > these web server instances and listen to port 80. Therefore it's the
| >> > only one who gets the request from clients and dispatch the request
to
| >> > one of the web servers who is relatively free base on certain
algorithm
| >> > and then pass its response to client during the same session;
| >> > 4. So everything is running *independantly* on the same machine but
we
| >> > can take advantage of multi-core processor and do some load balancing
| >> > work;
| >> >
| >> > Since I dont have any experience on implementing this kind of
service.
| >> > Could anybody kindly give me some hints?
| >> >
| >> > I'd specially appreciate Information like
| >> >
| >> > 1. What kind of Win32 technology do I need to know to be able to do
| >> > this stuff?
| >> > 2. How should the whole system work in term of socket / thread / I/O?
| >> > 3. Are there any existing products / source code / web sites which
can
| >> > do the same/similar thing?
| >> >
| >> > Thanks for your time!
| >> >
| >
|
|

a***@gmail.com

2006-10-12 05:35:44 UTC

Permalink

Hi Alex, thanks a lot for your suggestion. Actually I'd love to make
the web server as mutli-threaded if I can in the first place. But the
'core' processing component is not our product and it's really an old
fashion stuff since it's written more than 10 years ago, but
unfortunately, I'd say it's almost impossible to find a silimar
component now on the market (sorry I can't talk too much about it for
a obvious reason). Therefore, we have to design the web server as
single-threaded server to work with this thing. But I'll take your idea
about over-lapped I/O, I/O Completion Port and Thread-Pool to implement
this 'special' proxy and I think this should clear my question. Thanks
again for your time and input!

Post by Alexander Nickolov
Ah, so it's not an off-the-shelf web server, but something you
wrote, correct? In that case, may I suggest that the effort to
write the proxy may be better spent redesigning your server
to be multi-threaded? You'll likely want to change your socket
usage model as well. I'd guess you use async sockets (that's
the only somewhat viable model for a single-threaded server
as the other models are limited to 64 sockets). This doesn't
scale well due to the nature of the message loop.
At any rate, your original idea is a fairly straightforward
proxy server. You need to implement it efficiently of course,
so learn about overlapped socket I/O and I/O completion
ports (e.g. using a thread pool for overlapped I/O).
And just to be exhaustive, my original Netscaler proposal
would work for you too, though I admit it's probably an
overkill at this stage...
--
=====================================
Alexander Nickolov
Microsoft MVP [VC], MCSD
MVP VC FAQ: http://www.mvps.org/vcfaq
=====================================

Alexander Nickolov

2006-10-12 22:09:10 UTC

Permalink

You can isolate that legacy component into a standalone executable
and have your web server be the proxy. In fact, you may use an
off-the-shelf web server like IIS or Apache and simply interface
it with your legacy component... COM would be the logical choice
for interfacing with the legacy component, e.g. a single-threaded
single-use out-of-proc COM server. You'll create as many instances
of the interfacing COM object as you have CPU cores and share
them around (performing load balancing in the process of course).
--
=====================================
Alexander Nickolov
Microsoft MVP [VC], MCSD
email: ***@mvps.org
MVP VC FAQ: http://www.mvps.org/vcfaq
=====================================

Post by a***@gmail.com
Hi Alex, thanks a lot for your suggestion. Actually I'd love to make
the web server as mutli-threaded if I can in the first place. But the
'core' processing component is not our product and it's really an old
fashion stuff since it's written more than 10 years ago, but
unfortunately, I'd say it's almost impossible to find a silimar
component now on the market (sorry I can't talk too much about it for
a obvious reason). Therefore, we have to design the web server as
single-threaded server to work with this thing. But I'll take your idea
about over-lapped I/O, I/O Completion Port and Thread-Pool to implement
this 'special' proxy and I think this should clear my question. Thanks
again for your time and input!

Robert Zanatta

2006-10-13 01:25:02 UTC

Permalink

If you're using only one thread, use fibers along with io completion ports.

msgroup

2006-11-11 23:12:12 UTC

Permalink

Hi,

You may need to download the software package SocketPro at
www.udaparts.com. Particularly, you may need to play with the tutorial four
with documentation. That tutorial implements load balancing and directs all
of requests from many clients to different servers with batching, asynchrony
and parallel computation. I wish this technology catches your attention.

Regards,