实验目标：
1. LAMP节点提供用户动态请求访问，数据库单独有数据库节点提供；
2. LAMP动态网站有两台服务器，提供负载均衡；
3. 静态网站服务器节点提供用户的静态资源请求访问；存在两台静态web服务器，其网站静态资源在静态服务器上存放；
4. 用户的静态请求访问后缓存在varnish服务器上，实现访问加速
5. 前端的haproxy提供反向代理功能，将用户的动态资源请求发送给后端LAMP节点，静态资源请求发往后端静态web服务器；
6. 该架构考虑还不健全，如静态内容的一致性，数据库的单点故障，只是一个不成熟的架构实现简单的动静分离及缓存服务器实现；
实现过程：
一. LAMP构建
================ ha.stu31.com : 172.16.31.10 ================
vim /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
172.16.31.10 ha.stu31.com ha
172.16.31.11 node1.stu31.com node1
172.16.31.12 node2.stu31.com node2
172.16.31.13 node3.stu31.com node3
172.16.31.14 node4.stu31.com node4
172.16.31.15 mysql.stu31.com mysql
172.16.31.16 node5.stu31.com node5
172.16.31.17 node6.stu31.com node6
=======
ssh-keygen -t rsa -P "" -f "/root/.ssh/id_rsa"
for i in {1..6}; do ssh-copy-id -i .ssh/id_rsa.pub nodei;doneforiin1..6;doscp/etc/hostsnodei:/etc/hosts; done
for i in {1..6}; do ssh nodei"ntpdate−u202.112.10.60";doneforiin1..6;dosshnodei date; done
for i in {1..6}; do ssh nodei"mount/dev/cdrom/mnt/cdrom";doneforiin1..2;dosshnodei "yum -y install httpd php mysql php-mysql"; done
for i in {1..2}; do ssh nodei"servicehttpdstart";donevimindex.php<?phpphpinfo();?>///////////foriin1..2;doscpindex.phproot@nodei:/var/www/html; done
====================== windows ======================
浏览器访问 http://172.16.31.11/index.php
浏览器访问 http://172.16.31.12/index.php
============== mysql.stu31.com : 172.16.31.15 ================
mount /dev/cdrom /mnt/cdrom/
yum -y install mysql-server mysql
service mysqld start
mysql
mysql> create schema bbsdb;
mysql> grant all on bbsdb.* to 'bbsadmin'@'172.16.%.%'identified by 'oracle';
mysql> flush privileges;
mysql> quit
============== node1.stu31.com : 172.16.31.11 ================
wget http://download.comsenz.com/DiscuzX/3.2/Discuz_X3.2_SC_UTF8.zip
unzip -d /var/www/html/ Discuz_X3.2_SC_UTF8.zip
cd /var/www/html/upload
chmod -R go+w config/ data uc_*
====================== windows ======================
浏览器访问 http://172.16.31.11/upload/install
1. 检查安装环境：
2. 设置运行环境：点击选择全新安装
3. 安装数据库
填写数据库信息
数据库服务器: 172.16.31.15
数据库名: bbsdb
数据库用户名: bbsadmin
数据库密码: oracle
数据表前缀: pre_
系统邮箱: admin@stu31.com
填写管理员信息
管理员账号: admin
管理员密码: admin
重复密码: admin
管理员Email: admin@stu31.com
设置好数据库后点击安装即可
============== node2.stu31.com : 172.16.31.12 ===============
1. cd /var/www/html; scp -r upload/ readme/ utility/ node2:/var/www/html/
2. 重新解压软件包到node2，重新安装一遍，删除数据库重新安装(因为数据库中无重要内容)。
访问测试
====================== windows ======================
http://172.16.31.11/upload/forum.php
http://172.16.31.12/upload/forum.php
能正常访问证明LAMP平台构建成功
二. 静态web服务器构建
============== node5.stu31.com : 172.16.31.16 ===============
scp node1:/root/Discuz_X3.2_SC_UTF8.zip ./
(wget http://download.comsenz.com/DiscuzX/3.2/Discuz_X3.2_SC_UTF8.zip)
unzip -d /var/www/html/ Discuz_X3.2_SC_UTF8.zip
service httpd start
============== node6.stu31.com : 172.16.31.17 ===============
scp node1:/root/Discuz_X3.2_SC_UTF8.zip ./
(wget http://download.comsenz.com/DiscuzX/3.2/Discuz_X3.2_SC_UTF8.zip)
unzip -d /var/www/html/ Discuz_X3.2_SC_UTF8.zip
service httpd start
访问测试
====================== windows ======================
http://172.16.31.16/upload/forum.php
http://172.16.31.17/upload/forum.php
只能看到静态界面
三. Varnish缓存服务器构建
============== node3.stu31.com : 172.16.31.13 ===============
wget http://repo.varnish-cache.org/redhat/varnish-3.0/el6/x86_64/varnish/varnish-3.0.6-1.el6.x86_64.rpm
wget http://repo.varnish-cache.org/redhat/varnish-3.0/el6/x86_64/varnish/varnish-docs-3.0.6-1.el6.x86_64.rpm
wget http://repo.varnish-cache.org/redhat/varnish-3.0/el6/x86_64/varnish/varnish-libs-3.0.6-1.el6.x86_64.rpm
yum -y install *.rpm
vim /etc/sysconfig/varnish
NFILES=131072
MEMLOCK=82000
#NPROCS="unlimited" #加注释
DAEMON_COREFILE_LIMIT="unlimited" #去注释
RELOAD_VCL=1
VARNISH_VCL_CONF=/etc/varnish/default.vcl
VARNISH_LISTEN_PORT=80 #修改
VARNISH_ADMIN_LISTEN_ADDRESS=127.0.0.1
VARNISH_ADMIN_LISTEN_PORT=6082
VARNISH_SECRET_FILE=/etc/varnish/secret
VARNISH_MIN_THREADS=1 #修改
VARNISH_MAX_THREADS=1000
VARNISH_THREAD_TIMEOUT=120
VARNISH_STORAGE_FILE=/var/lib/varnish/varnish_storage.bin
VARNISH_STORAGE_SIZE=1G
VARNISH_STORAGE="file,VARNISHSTORAGEFILE,{VARNISH_STORAGE_SIZE}"
VARNISH_TTL=120
DAEMON_OPTS="-a VARNISHLISTENADDRESS:{VARNISH_LISTEN_PORT} \
-f VARNISHVCLCONF −T{VARNISH_ADMIN_LISTEN_ADDRESS}:VARNISHADMINLISTENPORT −t{VARNISH_TTL} \
-w VARNISHMINTHREADS,{VARNISH_MAX_THREADS},{VARNISH_THREAD_T -u varnish -g varnish \ -S{VARNISH_SECRET_FILE} \
-s {VARNISH_STORAGE}" /////////////////////// ( grep -v ^# /etc/sysconfig/varnish |sed '/^/d' )
cp /etc/varnish/default.vcl{,.bak}
vim /etc/varnish/default.vcl
:set nu
:15,s@^#@@g backend default { .host = "127.0.0.1"; .port = "80"; } # # Below is a commented-out copy of the default VCL logic. If you # redefine any of these subroutines, the built-in logic will be # appended to your code. sub vcl_recv { if (req.restarts == 0) { if (req.http.x-forwarded-for) { set req.http.X-Forwarded-For = req.http.X-Forwarded-For + ", " + client.ip; } else { set req.http.X-Forwarded-For = client.ip; } } if (req.request != "GET" && req.request != "HEAD" && req.request != "PUT" && req.request != "POST" && req.request != "TRACE" && req.request != "OPTIONS" && req.request != "DELETE") { /* Non-RFC2616 or CONNECT which is weird. */ return (pipe); } if (req.request != "GET" && req.request != "HEAD") { /* We only deal with GET and HEAD by default */ return (pass); } if (req.http.Authorization || req.http.Cookie) { /* Not cacheable by default */ return (pass); } return (lookup); } sub vcl_pipe { # Note that only the first request to the backend will have # X-Forwarded-For set. If you use X-Forwarded-For and want to # have it set for all requests, make sure to have: # set bereq.http.connection = "close"; # here. It is not set by default as it might break some broken web # applications, like IIS with NTLM authentication. return (pipe); } sub vcl_pass { return (pass); } sub vcl_hash { hash_data(req.url); if (req.http.host) { hash_data(req.http.host); } else { hash_data(server.ip); } return (hash); } sub vcl_hit { return (deliver); } sub vcl_miss { return (fetch); } sub vcl_fetch { if (beresp.ttl <= 0s || beresp.http.Set-Cookie || beresp.http.Vary == "*") { /* * Mark as "Hit-For-Pass" for the next 2 minutes */ set beresp.ttl = 120 s; return (hit_for_pass); } return (deliver); } sub vcl_deliver { return (deliver); } sub vcl_error { set obj.http.Content-Type = "text/html; charset=utf-8"; set obj.http.Retry-After = "5"; synthetic {" <?xml version="1.0" encoding="utf-8"?> <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"> <html> <head> <title>"} + obj.status + " " + obj.response + {"</title> </head> <body> <h1>Error "} + obj.status + " " + obj.response + {"</h1> <p>"} + obj.response + {"</p> <h3>Guru Meditation:</h3> <p>XID: "} + req.xid + {"</p> <hr> <p>Varnish cache server</p> </body> </html> "}; return (deliver); } sub vcl_init { return (ok); } sub vcl_fini { return (ok); } /////////////////// vim /etc/varnish/web.vcl :set paste ######定义ACL acl purgers { #定义acl，实现IP地址过滤 "127.0.0.1"; "172.16.0.0"/16; } ######定义健康状态检测 probe dynamic { #设置动态网站服务器健康状态检测 .url = "/index.html"; .interval = 5s; .timeout = 1s; .expected_response= 200; } #这里设置了两个健康状态检测主要是为了区分动、静网站 probe static { #设置动态网站服务器健康状态检测 .url = "/index.html"; #定义检测的页面 .interval = 5s; #探测请求的发送周期，默认为5秒 .timeout = 1s; #每次探测请求的过期时间 .expected_response = 200; } ######定义后端服务器 backend app1 { #定义一个后端服务器 .host = "172.16.31.11"; #服务器地址 .port = "80"; #服务器监听端口 .probe = dynamic; #健康状态检测 } backend app2 { .host = "172.16.31.12"; .port = "80"; .probe = dynamic; } backend web1 { .host = "172.16.31.16"; .port = "80"; .probe = static; } backend web2 { .host = "172.16.31.17"; .port = "80"; .probe = static; } ######定义后端动态服务器组，实现负载均衡效果 director apps random { #定义一个后端服务器组，实现负载均衡效果 { .backend = app1; #调用前面已定义过的后端主机 .weight = 2; #设置权重 } { .backend = app2; .weight = 2; } } ######定义后端静态服务器组，实现负载均衡效果 director webs random { { .backend = web1; .weight = 2 ; } { .backend = web2; .weight = 2 ; } } ######定义vcl_recv函数，实现请求到达并成功接收后调用此函数中定义的规则 sub vcl_recv { ######定义动、静分离,以".php"或".php?后面跟所有文件"结尾的请求都发送到动态服务器，其他请求都发送到静态服务器 if (req.url ~ "\.php(\?\.*|)") {
set req.backend = apps;
}else {
set req.backend = webs;
}
return(lookup);
######定义允许清除缓存的IP地址,调用的是前面定义的ACL
if (req.request == "PURGE") {
if (!client.ip ~ purgers) {
error 405 "Method not allowed";
}
return(lookup);
}
######重新定义http请求首部，让后端服务器可以记录请求客户端的真实IP地址
if (req.restarts == 0) {
if (req.http.x-forwarded-for) {
set req.http.X-Forwarded-For =
req.http.X-Forwarded-For +", " + client.ip;
} else {
set req.http.X-Forwarded-For =client.ip;
}
}
######除了定义的请求方法外，其他请求都到后端服务器
if (req.request != "GET" &&
req.request != "HEAD" &&
req.request != "PUT" &&
req.request != "POST" &&
req.request != "TRACE" &&
req.request != "OPTIONS" &&
req.request != "DELETE") {
return (pipe);
}
if (req.request != "GET" && req.request !="HEAD") {
return (pass);
}
######定义不缓存认证与Cookie信息
if (req.http.Authorization || req.http.Cookie) {
return (pass);
}
######定义压缩功能
if (req.http.Accept-Enconding) {
if (req.url ~ "\.(jpg|jpeg|gif|bmp|png|flv|gz|tgz|tbz|mp3)"){ remove req.http.Accept-Encoding; remove req.http.Cookie; } else if (req.http.Accept-Encoding ~ "gzip") { set req.http.Accept-Encoding = "gzip"; } else if (req.http.Accept-Encoding ~ "deflate") { set req.http.Accept-Encoding = "deflate"; } else { remove req.http.Accept-Encoding; } } ######定义指定格式结尾的文件去除Cookie信息 if (req.request == "GET" && req.url ~"\.(jpeg|jpg|gif|png|bmp|swf)") {
unset req.http.cookie;
}
######定义防盗链设置
if (req.http.referer ~ "http://.*") {
if (!(req.http.referer ~ "http://.*\.baidu\.com" ||req.http.referer ~ "http://.*\.google\.com.*")) {
set req.http.host ="www.stu31.com";
set req.url = "http://172.16.31.10/error.html";
}
}
}
######定义vcl_hash函数
sub vcl_hash {
hash_data(req.url);
if (req.http.host) {
hash_data(req.http.host);
}else {
hash_data(server.ip);
}
return(hash);
}
######定义vcl_hit函数
sub vcl_hit {
if (req.request == "PURGE") { #语法方法为"PURGE"
purge; #清除缓存
error 200 "Purged."; #返回错误状态码为"200"
}
return(deliver);
}
######定义vcl_miss函数
sub vcl_miss {
if (req.request == "PURGE") {
purge;
error 404 "Not In Cache.";
}
return(fetch);
}
######定义vcl_psss函数
sub vcl_pass {
if (req.request == "PURGE") {
error 502 "Purged On A Passed Object.";
}
return(pass);
}
######定义vcl_fetch函数
sub vcl_fetch {
######定义缓存，如果匹配到已定义文件结尾的缓存1天，其他则缓存1小时
if (req.request == "GET" && req.url ~"\.(html|jpg|png|bmp|jpeg|gif|js|ico|swf|css)$") {
set beresp.ttl = 1d;
set beresp.http.expires = beresp.ttl;
}else {
set beresp.ttl = 1h;
}
return(deliver);
}
######定义在http首部中，如果请求命中显示"HIT",未命中则显示"MISS",通过F12可查看缓存命中状态及varnish服务器IP
sub vcl_deliver {
if (obj.hits > 0) {
set resp.http.X-Cache = "Hit from " + server.ip;
}else {
set resp.http.X-Cache = "MISS";
}
}
//////////////////////////// end for /etc/varnish/web.vcl
/etc/rc.d/init.d/varnish start
netstat -tnlp | grep varnish
varnishadm -S/etc/varnish/secret -T 127.0.0.1:6082
varnish> vcl.load cache web.vcl
varnish> vcl.use cache
varnish> vcl.list
varnish> quit
============== node4.stu31.com : 172.16.31.14 ===============
scp node3:/root/*.rpm ./
yum -y install *.rpm
scp node3:/etc/sysconfig/varnish /etc/sysconfig/
scp node3:/etc/varnish/*.vcl /etc/varnish
/etc/rc.d/init.d/varnish start
netstat -tnlp | grep varnish
varnishadm -S/etc/varnish/secret -T 127.0.0.1:6082
varnish> vcl.load cache web.vcl
varnish> vcl.use cache #立刻生效, 重启varnish失效
varnish> vcl.list
varnish> quit
如果我们希望后端的web服务器记录客户端访问的真实IP地址，我们需要配置httpd的配置文件中的日志格式：
node1, node2, node5, node6四个节点都要
vim /etc/httpd/conf/httpd.conf
LogFormat "%{X-Forwarded-For}i %l %u%t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\""combined
///////////
service httpd reload
echo "ok" > /var/www/html/index.html #此页面用于varnish做健康检测
四. HAProxy反向代理构建
=============== ha.stu31.com : 172.16.31.10 ================
mount /dev/cdrom /mnt/cdrom
yum -y install haproxy
vim /etc/rsyslog.conf
# Save boot messages also to boot.log
local7.* /var/log/boot.log
local2.* /var/log/haproxy.log #添加此行
//////////////
service rsyslog restart
cd /etc/haproxy/
cp haproxy.cfg{,.bak}
vim haproxy.cfg
global
log 127.0.0.1 local2
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 4000
user haproxy
group haproxy
daemon
stats socket /var/lib/haproxy/stats
defaults
mode http
log global
option httplog
option dontlognull
option http-server-close
option forwardfor except 127.0.0.0/8
option redispatch
retries 3
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout http-keep-alive 10s
timeout check 10s
maxconn 3000
:set paste
listen stats #添加此块
mode http
bind 0.0.0.0:1080
stats enable
stats hide-version
stats uri /haproxyadmin?stats
stats realm Haproxy\ Statistics
stats auth admin:admin
stats admin if TRUE
frontend http-in #修改很多内容
bind *:80
mode http
log global
option httpclose
option logasap
option dontlognull
capture request header Host len 20
capture request header Referer len 60
acl url_static path_beg -i /upload/static/image/
acl url_static path_end -i .html .jpeg .gif .png .jpg
acl url_dynamic path_end -i .php .css .js .jsp
use_backend static_servers if url_static
use_backend dynamic_servers if url_dynamic
default_backend static_servers
backend static_servers
balance roundrobin
server staticsrv1 172.16.31.13:80 check maxconn 3000
server staticsrv2 172.16.31.14:80 check maxconn 3000
backend dynamic_servers
balance source
server dynamicsrv1 172.16.31.11:80 check maxconn 3000
server dynamicsrv2 172.16.31.12:80 check maxconn 3000
//////////////////
service haproxy start
五. 测试访问
====================== windows =====================
访问论坛
http://172.16.31.10/upload/forum.php
访问haproxy的状态页
http://172.16.31.10:1080/haproxyadmin?stats
admin
admin
window浏览器F12检查缓存命中情况
varnish上的日志
[root@node3 ~]# varnishlog
原始服务器的日志
[root@node1 ~]# tail /var/log/httpd/access_log
