crawlergo 動態爬蟲源碼學習
VSole2021-09-24 18:14:50
crawlergo是一個使用chrome headless模式進行URL收集的瀏覽器爬蟲。它對整個網頁的關鍵位置與DOM渲染階段進行HOOK，自動進行表單填充并提交，配合智能的JS事件觸發，盡可能的收集網站暴露出的入口。內置URL去重模塊，過濾掉了大量偽靜態URL，對于大型網站仍保持較快的解析與抓取速度，最后得到高質量的請求結果集合。crawlergo 目前支持以下特性：
* 原生瀏覽器環境，協程池調度任務
* 表單智能填充、自動化提交
* 完整DOM事件收集，自動化觸發
* 智能URL去重，去掉大部分的重復請求
* 全面分析收集，包括javascript文件內容、頁面注釋、robots.txt文件和常見路徑Fuzz
* 支持Host綁定，自動添加Referer
* 支持請求代理，支持爬蟲結果主動推送
Github: https://github.com/Qianlitp/crawlergo
作者開源了源碼，我是很興奮的，以前也有寫一個的想法，但是開源的動態爬蟲不多，看了其中幾個。
調研
1.https://github.com/fcavallarin/htcap
?遞歸dom搜索引擎
?發現ajax/fetch/jsonp/websocket請求
?支持cookie，代理，ua，http auth
?基于文本相似度的頁面重復數據刪除引擎
?根據文本長度 <256
?simhash
?else
?ShinglePrint
?主要代碼是python調用puppeteer，但是核心邏輯在js里
2.https://github.com/go-rod/rod
?一個操作chrome headless的go庫
?它比官方提供的chrome操作庫更容易使用
?有效解決了chrome殘留僵尸進程的問題
3.https://github.com/lc/gau
?通過一些通用接口獲取url信息
4.https://github.com/jaeles-project/gospider
?Web靜態爬蟲，也提供了一些方法獲取更多URL
5.https://github.com/chaitin/rad
1.rad雖然沒有開源，但是它里面使用yaml進行的配置選項很多，通過配置選項可以大致知道它的一些特性。
2.可以手動登陸
3.啟用圖片
4.顯示對爬取url的一些限制
1.不允許的文件后綴
2.不允許的url關鍵字
3.不允許的域名
4.不允許的url
5.設置下個頁面最大點擊和事件觸發
Crawlergo
之前也想過寫一個動態爬蟲來對接掃描器，但是動態爬蟲有很多細節都需要打磨，一直沒時間做，現在有現成的源碼參考能省下不少事。
主要看幾個點
?對瀏覽器 JavaScript環境的hoook
?dom的觸發，表單填充
?url如何去重
?url的收集
目錄結構
├─cmd
│ └─crawlergo # 程序主入口
├─examples
├─imgs
└─pkg
├─config # 一些配置相關
├─engine # chrome相關程序
├─filter # 去重相關
├─js # 一些注入的js
├─logger # 日志
├─model # url和請求相關的庫
└─tools # 一些通用類庫
└─requests
根據源碼的調用堆棧做了一個程序啟動流程圖
配置文件
pkg/config/config.go定義了一些默認的配置文件
const (
DefaultUA = "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.0 Safari/537.36"
MaxTabsCount = 10
TabRunTimeout = 20 * time.Second
DefaultInputText = "Crawlergo" // 默認輸入的文字
FormInputKeyword = "Crawlergo" // form輸入的文字，但是代碼中沒有引用這個變量的
SuspectURLRegex = `(?:"|')(((?:[a-zA-Z]{1,10}://|//)[^"'/]{1,}\.[a-zA-Z]{2,}[^"']{0,})|((?:/|\.\./|\./)[^"'><,;|*()(%%$^/\\\[\]][^"'><,;|()]{1,})|([a-zA-Z0-9_\-/]{1,}/[a-zA-Z0-9_\-/]{1,}\.(?:[a-zA-Z]{1,4}|action)(?:[\?|#][^"|']{0,}|))|([a-zA-Z0-9_\-/]{1,}/[a-zA-Z0-9_\-/]{3,}(?:[\?|#][^"|']{0,}|))|([a-zA-Z0-9_\-]{1,}\.(?:php|asp|aspx|jsp|json|action|html|js|txt|xml)(?:[\?|#][^"|']{0,}|)))(?:"|')` // url獲取正則
URLRegex = `((https?|ftp|file):)?//[-A-Za-z0-9+&@#/%?=~_|!:,.;]+[-A-Za-z0-9+&@#/%=~_|]` // url獲取正則
AttrURLRegex = ``
DomContentLoadedTimeout = 5 * time.Second
EventTriggerInterval = 100 * time.Millisecond // 單位毫秒
BeforeExitDelay = 1 * time.Second
DefaultEventTriggerMode = EventTriggerAsync
MaxCrawlCount = 200
)
// 請求的來源,記錄了每個url的來源，可以根據這些關鍵詞定位到相關的獲取代碼
const (
FromTarget = "Target" //初始輸入的目標
FromNavigation = "Navigation" //頁面導航請求
FromXHR = "XHR" //ajax異步請求
FromDOM = "DOM" //dom解析出來的請求
FromJSFile = "JavaScript" //JS腳本中解析
FromFuzz = "PathFuzz" //初始path fuzz
FromRobots = "robots.txt" //robots.txt
FromComment = "Comment" //頁面中的注釋
FromWebSocket = "WebSocket"
FromEventSource = "EventSource"
FromFetch = "Fetch"
FromHistoryAPI = "HistoryAPI"
FromOpenWindow = "OpenWindow"
FromHashChange = "HashChange"
FromStaticRes = "StaticResource"
FromStaticRegex = "StaticRegex"
)
// 靜態文件后綴，這些后綴的文件全部阻斷讀取
var StaticSuffix = []string{
"png", "gif", "jpg", "mp4", "mp3", "mng", "pct", "bmp", "jpeg", "pst", "psp", "ttf",
"tif", "tiff", "ai", "drw", "wma", "ogg", "wav", "ra", "aac", "mid", "au", "aiff",
"dxf", "eps", "ps", "svg", "3gp", "asf", "asx", "avi", "mov", "mpg", "qt", "rm",
"wmv", "m4a", "bin", "xls", "xlsx", "ppt", "pptx", "doc", "docx", "odt", "ods", "odg",
"odp", "exe", "zip", "rar", "tar", "gz", "iso", "rss", "pdf", "txt", "dll", "ico",
"gz2", "apk", "crt", "woff", "map", "woff2", "webp", "less", "dmg", "bz2", "otf", "swf",
"flv", "mpeg", "dat", "xsl", "csv", "cab", "exif", "wps", "m4v", "rmvb",
}
// 動態文件的后綴，過濾器用于過濾偽靜態用
var ScriptSuffix = []string{
"php", "asp", "jsp", "asa",
}
// 默認不爬的url關鍵詞
var DefaultIgnoreKeywords = []string{"logout", "quit", "exit"}
// 填充表單相關，這些是模糊匹配，匹配到了則使用對應的規則
var AllowedFormName = []string{"default", "mail", "code", "phone", "username", "password", "qq", "id_card", "url", "date", "number"}
var InputTextMap = map[string]map[string]interface{}{
"mail": {
"keyword": []string{"mail"},
"value": "crawlergo@gmail.com",
},
"code": {
"keyword": []string{"yanzhengma", "code", "ver", "captcha"},
"value": "123a",
},
"phone": {
"keyword": []string{"phone", "number", "tel", "shouji"},
"value": "18812345678",
},
"username": {
"keyword": []string{"name", "user", "id", "login", "account"},
"value": "crawlergo@gmail.com",
},
"password": {
"keyword": []string{"pass", "pwd"},
"value": "Crawlergo6.",
},
"qq": {
"keyword": []string{"qq", "wechat", "tencent", "weixin"},
"value": "123456789",
},
"IDCard": {
"keyword": []string{"card", "shenfen"},
"value": "511702197409284963",
},
"url": {
"keyword": []string{"url", "site", "web", "blog", "link"},
"value": "https://crawlergo.nice.cn/",
},
"date": {
"keyword": []string{"date", "time", "year", "now"},
"value": "2018-01-01",
},
"number": {
"keyword": []string{"day", "age", "num", "count"},
"value": "10",
},
}
URL過濾方式
crawlergo有兩種過濾，fensimple和smart。
simple
simple過濾方式比較簡單，就是將計算請求體的method、url、postdata 結合計算md5
判斷是否存在
smart
smart過濾會對每個請求的參數name，參數value，path 進行標記,會標記成以下字段
const (
CustomValueMark = "{{Crawlergo}}"
FixParamRepeatMark = "{{fix_param}}"
FixPathMark = "{{fix_path}}"
TooLongMark = "{{long}}"
NumberMark = "{{number}}"
ChineseMark = "{{chinese}}"
UpperMark = "{{upper}}"
LowerMark = "{{lower}}"
UrlEncodeMark = "{{urlencode}}"
UnicodeMark = "{{unicode}}"
BoolMark = "{{bool}}"
ListMark = "{{list}}"
TimeMark = "{{time}}"
MixAlphaNumMark = "{{mix_alpha_num}}"
MixSymbolMark = "{{mix_symbol}}"
MixNumMark = "{{mix_num}}"
NoLowerAlphaMark = "{{no_lower}}"
MixStringMark = "{{mix_str}}"
)
例如
?m=home&c=index&a=index ?type=202cb962ac59075b964b07152d234b70 ?id=1 ?msg=%E6%B6%88%E6%81%AF
處理結果即
?m=home&c=index&a=index
?type={hash}
?id={int}
?msg={urlencode}
會對超過閾值的結果，以及計算的唯一id進行比對，來判斷是否過濾，總體來說是基于url的過濾方式。
使用的正則
var chineseRegex = regexp.MustCompile("[\u4e00-\u9fa5]+")
var urlencodeRegex = regexp.MustCompile("(?:%[A-Fa-f0-9]{2,6})+")
var unicodeRegex = regexp.MustCompile(`(?:\\u\w{4})+`)
var onlyAlphaRegex = regexp.MustCompile("^[a-zA-Z]+$")
var onlyAlphaUpperRegex = regexp.MustCompile("^[A-Z]+$")
var alphaUpperRegex = regexp.MustCompile("[A-Z]+")
var alphaLowerRegex = regexp.MustCompile("[a-z]+")
var replaceNumRegex = regexp.MustCompile(`[0-9]+\.[0-9]+|\d+`)
var onlyNumberRegex = regexp.MustCompile(`^[0-9]+$`)
var numberRegex = regexp.MustCompile(`[0-9]+`)
var OneNumberRegex = regexp.MustCompile(`[0-9]`)
var numSymbolRegex = regexp.MustCompile(`\.|_|-`)
var timeSymbolRegex = regexp.MustCompile(`-|:|\s`)
var onlyAlphaNumRegex = regexp.MustCompile(`^[0-9a-zA-Z]+$`)
var markedStringRegex = regexp.MustCompile(`^{{.+}}$`)
var htmlReplaceRegex = regexp.MustCompile(`\.shtml|\.html|\.htm`)
對路徑的處理
/**
標記路徑
*/
func (s *SmartFilter) MarkPath(path string) string {
pathParts := strings.Split(path, "/")
for index, part := range pathParts {
if len(part) >= 32 {
pathParts[index] = TooLongMark
} else if onlyNumberRegex.MatchString(numSymbolRegex.ReplaceAllString(part, "")) {
pathParts[index] = NumberMark
} else if strings.HasSuffix(part, ".html") || strings.HasSuffix(part, ".htm") || strings.HasSuffix(part, ".shtml") {
part = htmlReplaceRegex.ReplaceAllString(part, "")
// 大寫、小寫、數字混合
if numberRegex.MatchString(part) && alphaUpperRegex.MatchString(part) && alphaLowerRegex.MatchString(part) {
pathParts[index] = MixAlphaNumMark
// 純數字
} else if b := numSymbolRegex.ReplaceAllString(part, ""); onlyNumberRegex.MatchString(b) {
pathParts[index] = NumberMark
}
// 含有特殊符號
} else if s.hasSpecialSymbol(part) {
pathParts[index] = MixSymbolMark
} else if chineseRegex.MatchString(part) {
pathParts[index] = ChineseMark
} else if unicodeRegex.MatchString(part) {
pathParts[index] = UnicodeMark
} else if onlyAlphaUpperRegex.MatchString(part) {
pathParts[index] = UpperMark
// 均為數字和一些符號組成
} else if b := numSymbolRegex.ReplaceAllString(part, ""); onlyNumberRegex.MatchString(b) {
pathParts[index] = NumberMark
// 數字出現的次數超過3，視為偽靜態path
} else if b := OneNumberRegex.ReplaceAllString(part, "0"); strings.Count(b, "0") > 3 {
pathParts[index] = MixNumMark
}
}
newPath := strings.Join(pathParts, "/")
return newPath
}
基于網頁結構去重
作者原帖中的基于網頁結構去重寫的非常精彩 https://www.anquanke.com/post/id/178339#h2-17
參考的論文下載: https://patents.google.com/patent/CN101694668B/zh
作者的將網頁特征向量抽離出來，索引存儲，用來判斷大量網頁的相似度，非常驚艷，未來用到資產收集系統或者網絡空間引擎上也都是非常不錯的選擇。
URL收集
robots
在爬蟲之前，會先請求robots.txt，解析出所有鏈接，加入到待爬取頁面。
源碼中使用了一個正則來匹配
var urlFindRegex = regexp.MustCompile(`(?:Disallow|Allow):.*?(/.+)`)
看了下robots規范:https://baike.baidu.com/item/robots%E5%8D%8F%E8%AE%AE/2483797 ，應該還可以再優化一下，來處理一些表達式。
DIR FUZZ
在爬蟲之前，如果沒有指定dir字典的話，默認會使用內置的字典
['11', '123', '2017', '2018', 'message', 'mis', 'model', 'abstract', 'account', 'act', 'action', 'activity', 'ad', 'address', 'ajax', 'alarm', 'api', 'app', 'ar', 'attachment', 'auth', 'authority', 'award', 'back', 'backup', 'bak', 'base', 'bbs', 'bbs1', 'cms', 'bd', 'gallery', 'game', 'gift', 'gold', 'bg', 'bin', 'blacklist', 'blog', 'bootstrap', 'brand', 'build', 'cache', 'caches', 'caching', 'cacti', 'cake', 'captcha', 'category', 'cdn', 'ch', 'check', 'city', 'class', 'classes', 'classic', 'client', 'cluster', 'collection', 'comment', 'commit', 'common', 'commons', 'components', 'conf', 'config', 'mysite', 'confs', 'console', 'consumer', 'content', 'control', 'controllers', 'core', 'crontab', 'crud', 'css', 'daily', 'dashboard', 'data', 'database', 'db', 'default', 'demo', 'dev', 'doc', 'download', 'duty', 'es', 'eva', 'examples', 'excel', 'export', 'ext', 'fe', 'feature', 'file', 'files', 'finance', 'flashchart', 'follow', 'forum', 'frame', 'framework', 'ft', 'group', 'gss', 'hello', 'helper', 'helpers', 'history', 'home', 'hr', 'htdocs', 'html', 'hunter', 'image', 'img11', 'import', 'improve', 'inc', 'include', 'includes', 'index', 'info', 'install', 'interface', 'item', 'jobconsume', 'jobs', 'json', 'kindeditor', 'l', 'languages', 'lib', 'libraries', 'libs', 'link', 'lite', 'local', 'log', 'login', 'logs', 'mail', 'main', 'maintenance', 'manage', 'manager', 'manufacturer', 'menus', 'models', 'modules', 'monitor', 'movie', 'mysql', 'n', 'nav', 'network', 'news', 'notice', 'nw', 'oauth', 'other', 'page', 'pages', 'passport', 'pay', 'pcheck', 'people', 'person', 'php', 'phprpc', 'phptest', 'picture', 'pl', 'platform', 'pm', 'portal', 'post', 'product', 'project', 'protected', 'proxy', 'ps', 'public', 'qq', 'question', 'quote', 'redirect', 'redisclient', 'report', 'resource', 'resources', 's', 'save', 'schedule', 'schema', 'script', 'scripts', 'search', 'security', 'server', 'service', 'shell', 'show', 'simple', 'site', 'sites', 'skin', 'sms', 'soap', 'sola', 'sort', 'spider', 'sql', 'stat', 'static', 'statistics', 'stats', 'submit', 'subways', 'survey', 'sv', 'syslog', 'system', 'tag', 'task', 'tasks', 'tcpdf', 'template', 'templates', 'test', 'tests', 'ticket', 'tmp', 'token', 'tool', 'tools', 'top', 'tpl', 'txt', 'upload', 'uploadify', 'uploads', 'url', 'user', 'util', 'v1', 'v2', 'vendor', 'view', 'views', 'web', 'weixin', 'widgets', 'wm', 'wordpress', 'workspace', 'ws', 'www', 'www2', 'wwwroot', 'zone', 'admin', 'admin_bak', 'mobile', 'm', 'js']
根據狀態碼判斷
爬蟲時的url收集
?解析流量中的url
?獲取xmr類型的請求
?正則解析js、html、json中url
?收集當前頁面上的url信息
"src", "href", "data-url", "data-href" object[data] 注釋中的url
對瀏覽器環境的hook
在chrome的tab初始化時，會執行一段js代碼，hook部分函數和事件來控制js的運行環境。
同時定義了一個js全局函數addLink、Test，通過這個函數可以與go進行交互。
調用go函數
js初始化時對這個函數重新包裝
const binding = window["addLink"];
window["addLink"] = async(...args) => {
const me = window["addLink"];
let callbacks = me['callbacks'];
if (!callbacks) {
callbacks = new Map();
me['callbacks'] = callbacks;
}
const seq = (me['lastSeq'] || 0) + 1;
me['lastSeq'] = seq;
const promise = new Promise(fulfill => callbacks.set(seq, fulfill));
binding(JSON.stringify({name: "addLink", seq, args}));
return promise;
};
const bindingTest = window["Test"];
window["Test"] = async(...args) => {
const me = window["Test"];
let callbacks = me['callbacks'];
if (!callbacks) {
callbacks = new Map();
me['callbacks'] = callbacks;
}
const seq = (me['lastSeq'] || 0) + 1;
me['lastSeq'] = seq;
const promise = new Promise(fulfill => callbacks.set(seq, fulfill));
binding(JSON.stringify({name: "Test", seq, args}));
return promise;
};
go處理邏輯
執行完go函數后會再執行一段js
const DeliverResultJS = `
(function deliverResult(name, seq, result) {
window[name]['callbacks'].get(seq)(result);
window[name]['callbacks'].delete(seq);
})("%s", %v, "%s")
但是沒看懂使用promise后回調調用的意義是什么。。
Bypass headless detect
// Pass the Webdriver Test.
Object.defineProperty(navigator, 'webdriver', {
get: () => false,
});
// Pass the Plugins Length Test.
// Overwrite the plugins property to use a custom getter.
Object.defineProperty(navigator, 'plugins', {
// This just needs to have length > 0 for the current test,
// but we could mock the plugins too if necessary.
get: () => [1, 2, 3, 4, 5],
});
// Pass the Chrome Test.
// We can mock this in as much depth as we need for the test.
window.chrome = {
runtime: {},
};
// Pass the Permissions Test.
const originalQuery = window.navigator.permissions.query;
window.navigator.permissions.query = (parameters) => (
parameters.name === 'notifications' ?
Promise.resolve({ state: Notification.permission }) :
originalQuery(parameters)
);
//Pass the Permissions Test. navigator.userAgent
Object.defineProperty(navigator, 'userAgent', {
get: () => "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.0 Safari/537.36",
});
// 修改瀏覽器對象的屬性
Object.defineProperty(navigator, 'platform', {
get: function () { return 'win32'; }
});
Object.defineProperty(navigator, 'language', {
get: function () { return 'zh-CN'; }
});
Object.defineProperty(navigator, 'languages', {
get: function () { return ["zh-CN", "zh"]; }
});
記錄url以及對前端框架的適配
// history api hook 許多前端框架都采用此API進行頁面路由，記錄url并取消操作
window.history.pushState = function(a, b, c) {
window.addLink(c, "HistoryAPI");
}
window.history.replaceState = function(a, b, c) {
window.addLink(c, "HistoryAPI");
}
Object.defineProperty(window.history,"pushState",{"writable": false, "configurable": false});
Object.defineProperty(window.history,"replaceState",{"writable": false, "configurable": false});
// 監聽hash改變 Vue等框架默認使用hash部分進行前端頁面路由
window.addEventListener("hashchange", function() {
window.addLink(document.location.href, "HashChange");
});
// 監聽窗口的打開和關閉，記錄新窗口打開的url，并取消實際操作
// hook window.open
window.open = function (url) {
console.log("trying to open window.");
window.addLink(url, "OpenWindow");
}
Object.defineProperty(window,"open",{"writable": false, "configurable": false});
// hook window close
window.close = function() {console.log("trying to close page.");};
Object.defineProperty(window,"close",{"writable": false, "configurable": false});
// hook window.WebSocket 、window.EventSource 、 window.fetch 等函數
var oldWebSocket = window.WebSocket;
window.WebSocket = function(url, arg) {
window.addLink(url, "WebSocket");
return new oldWebSocket(url, arg);
}
var oldEventSource = window.EventSource;
window.EventSource = function(url) {
window.addLink(url, "EventSource");
return new oldEventSource(url);
}
var oldFetch = window.fetch;
window.fetch = function(url) {
window.addLink(url, "Fetch");
return oldFetch(url);
}
hook setTimeout/SetInterval
// hook setTimeout
//window.__originalSetTimeout = window.setTimeout;
//window.setTimeout = function() {
// arguments[1] = 0;
// return window.__originalSetTimeout.apply(this, arguments);
//};
//Object.defineProperty(window,"setTimeout",{"writable": false, "configurable": false});
// hook setInterval 時間設置為60秒 目的是減輕chrome的壓力
window.__originalSetInterval = window.setInterval;
window.setInterval = function() {
arguments[1] = 60000;
return window.__originalSetInterval.apply(this, arguments);
};
Object.defineProperty(window,"setInterval",{"writable": false, "configurable": false});
這個hook操作沒有明白，將setInterval強制設為了60s，我想應該來個判斷，大于60s時再統一設置，60s，這樣爬蟲效率就變得太低了。
setTimeout取消了hook，畢竟只執行一次，可能不是很重要。
hook ajax
hook 原生ajax并限制最大請求數，可能是怕自動點擊造成ajax爆炸
// 劫持原生ajax，并對每個請求設置最大請求次數
window.ajax_req_count_sec_auto = {};
XMLHttpRequest.prototype.__originalOpen = XMLHttpRequest.prototype.open;
XMLHttpRequest.prototype.open = function(method, url, async, user, password) {
// hook code
this.url = url;
this.method = method;
let name = method + url;
if (!window.ajax_req_count_sec_auto.hasOwnProperty(name)) {
window.ajax_req_count_sec_auto[name] = 1
} else {
window.ajax_req_count_sec_auto[name] += 1
}
if (window.ajax_req_count_sec_auto[name] <= 10) {
return this.__originalOpen(method, url, true, user, password);
}
}
Object.defineProperty(XMLHttpRequest.prototype,"open",{"writable": false, "configurable": false});
XMLHttpRequest.prototype.__originalSend = XMLHttpRequest.prototype.send;
XMLHttpRequest.prototype.send = function(data) {
// hook code
let name = this.method + this.url;
if (window.ajax_req_count_sec_auto[name] <= 10) {
return this.__originalSend(data);
}
}
Object.defineProperty(XMLHttpRequest.prototype,"send",{"writable": false, "configurable": false});
XMLHttpRequest.prototype.__originalAbort = XMLHttpRequest.prototype.abort;
XMLHttpRequest.prototype.abort = function() {
// hook code
}
Object.defineProperty(XMLHttpRequest.prototype,"abort",{"writable": false, "configurable": false});
鎖定表單重置
爬蟲在處理網頁時，會先填充表單，接著觸發事件去提交表單，但有時會意外點擊到表單的重置按鈕，造成內容清空，表單提交失敗。所以為了防止這種情況的發生，需要Hook表單的重置并鎖定不能修改。
// 鎖定表單重置
HTMLFormElement.prototype.reset = function() {console.log("cancel reset form")};
Object.defineProperty(HTMLFormElement.prototype,"reset",{"writable": false, "configurable": false});
事件Hook
對dom0級和dom2級事件分別hook，用一個數組設定最大觸發次數，所有js調用的事件，會對每個標簽設置一個sec_auto_dom2_event_flag屬性，方便后面尋找并自動觸發。
// hook dom2 級事件監聽
window.add_even_listener_count_sec_auto = {};
// record event func , hook addEventListener
let old_event_handle = Element.prototype.addEventListener;
Element.prototype.addEventListener = function(event_name, event_func, useCapture) {
let name = "<" + this.tagName + "> " + this.id + this.name + this.getAttribute("class") + "|" + event_name;
// console.log(name)
// 對每個事件設定最大的添加次數，防止無限觸發，最大次數為5
if (!window.add_even_listener_count_sec_auto.hasOwnProperty(name)) {
window.add_even_listener_count_sec_auto[name] = 1;
} else if (window.add_even_listener_count_sec_auto[name] == 5) {
return ;
} else {
window.add_even_listener_count_sec_auto[name] += 1;
}
if (this.hasAttribute("sec_auto_dom2_event_flag")) {
let sec_auto_dom2_event_flag = this.getAttribute("sec_auto_dom2_event_flag");
this.setAttribute("sec_auto_dom2_event_flag", sec_auto_dom2_event_flag + "|" + event_name);
} else {
this.setAttribute("sec_auto_dom2_event_flag", event_name);
}
old_event_handle.apply(this, arguments);
};
function dom0_listener_hook(that, event_name) {
let name = "<" + that.tagName + "> " + that.id + that.name + that.getAttribute("class") + "|" + event_name;
// console.log(name);
// 對每個事件設定最大的添加次數，防止無限觸發，最大次數為5
if (!window.add_even_listener_count_sec_auto.hasOwnProperty(name)) {
window.add_even_listener_count_sec_auto[name] = 1;
} else if (window.add_even_listener_count_sec_auto[name] == 5) {
return ;
} else {
window.add_even_listener_count_sec_auto[name] += 1;
}
if (that.hasAttribute("sec_auto_dom2_event_flag")) {
let sec_auto_dom2_event_flag = that.getAttribute("sec_auto_dom2_event_flag");
that.setAttribute("sec_auto_dom2_event_flag", sec_auto_dom2_event_flag + "|" + event_name);
} else {
that.setAttribute("sec_auto_dom2_event_flag", event_name);
}
}
// hook dom0 級事件監聽
Object.defineProperties(HTMLElement.prototype, {
onclick: {set: function(newValue){onclick = newValue;dom0_listener_hook(this, "click");}},
onchange: {set: function(newValue){onchange = newValue;dom0_listener_hook(this, "change");}},
onblur: {set: function(newValue){onblur = newValue;dom0_listener_hook(this, "blur");}},
ondblclick: {set: function(newValue){ondblclick = newValue;dom0_listener_hook(this, "dbclick");}},
onfocus: {set: function(newValue){onfocus = newValue;dom0_listener_hook(this, "focus");}},
onkeydown: {set: function(newValue){onkeydown = newValue;dom0_listener_hook(this, "keydown");}},
onkeypress: {set: function(newValue){onkeypress = newValue;dom0_listener_hook(this, "keypress");}},
onkeyup: {set: function(newValue){onkeyup = newValue;dom0_listener_hook(this, "keyup");}},
onload: {set: function(newValue){onload = newValue;dom0_listener_hook(this, "load");}},
onmousedown: {set: function(newValue){onmousedown = newValue;dom0_listener_hook(this, "mousedown");}},
onmousemove: {set: function(newValue){onmousemove = newValue;dom0_listener_hook(this, "mousemove");}},
onmouseout: {set: function(newValue){onmouseout = newValue;dom0_listener_hook(this, "mouseout");}},
onmouseover: {set: function(newValue){onmouseover = newValue;dom0_listener_hook(this, "mouseover");}},
onmouseup: {set: function(newValue){onmouseup = newValue;dom0_listener_hook(this, "mouseup");}},
onreset: {set: function(newValue){onreset = newValue;dom0_listener_hook(this, "reset");}},
onresize: {set: function(newValue){onresize = newValue;dom0_listener_hook(this, "resize");}},
onselect: {set: function(newValue){onselect = newValue;dom0_listener_hook(this, "select");}},
onsubmit: {set: function(newValue){onsubmit = newValue;dom0_listener_hook(this, "submit");}},
onunload: {set: function(newValue){onunload = newValue;dom0_listener_hook(this, "unload");}},
onabort: {set: function(newValue){onabort = newValue;dom0_listener_hook(this, "abort");}},
onerror: {set: function(newValue){onerror = newValue;dom0_listener_hook(this, "error");}},
})
表單填充,事件觸發
表單填充
input處理
// 找出 type 為空 或者 type=text
for _, node := range nodes {
// 兜底超時
tCtxN, cancelN := context.WithTimeout(ctx, time.Second*5)
attrType := node.AttributeValue("type")
if attrType == "text" || attrType == "" {
inputName := node.AttributeValue("id") + node.AttributeValue("class") + node.AttributeValue("name")
value := f.GetMatchInputText(inputName)
// 尋找匹配類型的值
var nodeIds = []cdp.NodeID{node.NodeID}
// 先使用模擬輸入
_ = chromedp.SendKeys(nodeIds, value, chromedp.ByNodeID).Do(tCtxN)
// 再直接賦值JS屬性
_ = chromedp.SetAttributeValue(nodeIds, "value", value, chromedp.ByNodeID).Do(tCtxN)
} else if attrType == "email" || attrType == "password" || attrType == "tel" {
value := f.GetMatchInputText(attrType)
// 尋找匹配類型的值
var nodeIds = []cdp.NodeID{node.NodeID}
// 先使用模擬輸入
_ = chromedp.SendKeys(nodeIds, value, chromedp.ByNodeID).Do(tCtxN)
// 再直接賦值JS屬性
_ = chromedp.SetAttributeValue(nodeIds, "value", value, chromedp.ByNodeID).Do(tCtxN)
} else if attrType == "radio" || attrType == "checkbox" {
var nodeIds = []cdp.NodeID{node.NodeID}
_ = chromedp.SetAttributeValue(nodeIds, "checked", "true", chromedp.ByNodeID).Do(tCtxN)
} else if attrType == "file" || attrType == "image" {
var nodeIds = []cdp.NodeID{node.NodeID}
wd, _ := os.Getwd()
filePath := wd + "/upload/image.png"
_ = chromedp.RemoveAttribute(nodeIds, "accept", chromedp.ByNodeID).Do(tCtxN)
_ = chromedp.RemoveAttribute(nodeIds, "required", chromedp.ByNodeID).Do(tCtxN)
// 對于一些簡單的限制，可以去掉，比如找到文件上傳的dom節點并刪除 accept 和 required 屬性：
_ = chromedp.SendKeys(nodeIds, filePath, chromedp.ByNodeID).Do(tCtxN)
}
cancelN()
}
multiSelect
css語法獲取select第一個元素，設置屬性即可
optionNodes, optionErr := f.tab.GetNodeIDs(`select option:first-child`)
if optionErr != nil || len(optionNodes) == 0 {
logger.Logger.Debug("fillMultiSelect: get select option element err")
if optionErr != nil {
logger.Logger.Debug(optionErr)
}
return
}
_ = chromedp.SetAttributeValue(optionNodes, "selected", "true", chromedp.ByNodeID).Do(tCtx)
_ = chromedp.SetJavascriptAttribute(optionNodes, "selected", "true", chromedp.ByNodeID).Do(tCtx)
TextArea
找到后填充即可
textareaNodes, textareaErr := f.tab.GetNodeIDs(`textarea`)
if textareaErr != nil || len(textareaNodes) == 0 {
logger.Logger.Debug("fillTextarea: get textarea element err")
if textareaErr != nil {
logger.Logger.Debug(textareaErr)
}
return
}
_ = chromedp.SendKeys(textareaNodes, value, chromedp.ByNodeID).Do(tCtx)
自動化提交表單
提交表單也有一些需要注意的問題，直接點擊form表單的提交按鈕會導致頁面重載，我們并不希望當前頁面刷新，所以除了Hook住前端導航請求之外，我們還可以為form節點設置target屬性，指向一個隱藏的iframe。具體操作的話就是新建隱藏iframe然后將form表單的target指向它即可
/**
設置form的target指向一個frame
*/
const NewFrameTemplate = `
(function sec_auto_new_iframe () {
let frame = document.createElement("iframe");
frame.setAttribute("name", "%s");
frame.setAttribute("id", "%s");
frame.setAttribute("style", "display: none");
document.body.appendChild(frame);
})()
`
func (tab *Tab) setFormToFrame() {
// 首先新建 frame
nameStr := tools.RandSeq(8)
tab.Evaluate(fmt.Sprintf(js.NewFrameTemplate, nameStr, nameStr))
// 接下來將所有的 form 節點target都指向它
ctx := tab.GetExecutor()
formNodes, formErr := tab.GetNodeIDs(`form`)
if formErr != nil || len(formNodes) == 0 {
logger.Logger.Debug("setFormToFrame: get form element err")
if formErr != nil {
logger.Logger.Debug(formErr)
}
return
}
tCtx, cancel := context.WithTimeout(ctx, time.Second*2)
defer cancel()
_ = chromedp.SetAttributeValue(formNodes, "target", nameStr, chromedp.ByNodeID).Do(tCtx)
}
要成功的提交表單，就得正確觸發表單的submit操作。不是所有的前端內容都有規范的表單格式，或許有一些form連個button都沒有，所以這里有三種思路可供嘗試，保險起見建議全部都運行一次：
?在form節點的子節點內尋找type=submit的節點，執行elementHandle.click()方法。
?直接對form節點執行JS語句：form.submit()，注意，如果form內有包含屬性值name=submit的節點，將會拋出異常，所以注意捕獲異常。
?在form節點的子節點內尋找所有button節點，全部執行一次elementHandle.click()方法。因為我們之前已經重定義并鎖定了表單重置函數，所以不用擔心會清空表單。
這樣，絕大部分表單我們都能觸發了。
/**
點擊按鈕 type=submit
*/
func (tab *Tab) clickSubmit() {
defer tab.formSubmitWG.Done()
// 首先點擊按鈕 type=submit
ctx := tab.GetExecutor()
// 獲取所有的form節點 直接執行submit
formNodes, formErr := tab.GetNodeIDs(`form`)
if formErr != nil || len(formNodes) == 0 {
logger.Logger.Debug("clickSubmit: get form element err")
if formErr != nil {
logger.Logger.Debug(formErr)
}
return
}
tCtx1, cancel1 := context.WithTimeout(ctx, time.Second*2)
defer cancel1()
_ = chromedp.Submit(formNodes, chromedp.ByNodeID).Do(tCtx1)
// 獲取所有的input標簽
inputNodes, inputErr := tab.GetNodeIDs(`form input[type=submit]`)
if inputErr != nil || len(inputNodes) == 0 {
logger.Logger.Debug("clickSubmit: get form input element err")
if inputErr != nil {
logger.Logger.Debug(inputErr)
}
return
}
tCtx2, cancel2 := context.WithTimeout(ctx, time.Second*2)
defer cancel2()
_ = chromedp.Click(inputNodes, chromedp.ByNodeID).Do(tCtx2)
}
/**
click all button
*/
func (tab *Tab) clickAllButton() {
defer tab.formSubmitWG.Done()
// 獲取所有的form中的button節點
ctx := tab.GetExecutor()
// 獲取所有的button標簽
btnNodeIDs, bErr := tab.GetNodeIDs(`form button`)
if bErr != nil || len(btnNodeIDs) == 0 {
logger.Logger.Debug("clickAllButton: get form button element err")
if bErr != nil {
logger.Logger.Debug(bErr)
}
return
}
tCtx, cancel1 := context.WithTimeout(ctx, time.Second*2)
defer cancel1()
_ = chromedp.Click(btnNodeIDs, chromedp.ByNodeID).Do(tCtx)
// 使用JS的click方法進行點擊
var btnNodes []*cdp.Node
tCtx2, cancel2 := context.WithTimeout(ctx, time.Second*2)
defer cancel2()
err := chromedp.Nodes(btnNodeIDs, &btnNodes, chromedp.ByNodeID).Do(tCtx2)
if err != nil {
return
}
for _, node := range btnNodes {
_ = tab.EvaluateWithNode(js.FormNodeClickJS, node)
}
}
事件觸發
1.對JavaScript協議的內聯事件觸發，執行以下js
(async function click_all_a_tag_javascript(){
let nodeListHref = document.querySelectorAll("[href]");
nodeListHref = window.randArr(nodeListHref);
for (let node of nodeListHref) {
let attrValue = node.getAttribute("href");
if (attrValue.toLocaleLowerCase().startsWith("javascript:")) {
await window.sleep(%f);
try {
eval(attrValue.substring(11));
}
catch {}
}
}
let nodeListSrc = document.querySelectorAll("[src]");
nodeListSrc = window.randArr(nodeListSrc);
for (let node of nodeListSrc) {
let attrValue = node.getAttribute("src");
if (attrValue.toLocaleLowerCase().startsWith("javascript:")) {
await window.sleep(%f);
try {
eval(attrValue.substring(11));
}
catch {}
}
}
})()
2.對常見的內聯事件觸發
(async function trigger_all_inline_event(){
let eventNames = ["onabort", "onblur", "onchange", "onclick", "ondblclick", "onerror", "onfocus", "onkeydown", "onkeypress", "onkeyup", "onload", "onmousedown", "onmousemove", "onmouseout", "onmouseover", "onmouseup", "onreset", "onresize", "onselect", "onsubmit", "onunload"];
for (let eventName of eventNames) {
let event = eventName.replace("on", "");
let nodeList = document.querySelectorAll("[" + eventName + "]");
if (nodeList.length > 100) {
nodeList = nodeList.slice(0, 100);
}
nodeList = window.randArr(nodeList);
for (let node of nodeList) {
await window.sleep(%f);
let evt = document.createEvent('CustomEvent');
evt.initCustomEvent(event, false, true, null);
try {
node.dispatchEvent(evt);
}
catch {}
}
}
})()
3.對之前hook的事件觸發，對于某些節點，可能會存在子節點也響應的事件，為了性能考慮，可以將層數控制到三層，且對兄弟節點隨機選擇一個觸發。簡單畫圖說明：
1.
(async function trigger_all_dom2_custom_event() {
function transmit_child(node, event, loop) {
let _loop = loop + 1
if (_loop > 4) {
return;
}
if (node.nodeType === 1) {
if (node.hasChildNodes) {
let index = parseInt(Math.random()*node.children.length,10);
try {
node.children[index].dispatchEvent(event);
} catch(e) {}
let max = node.children.length>5?5:node.children.length;
for (let count=0;count<max;count++) {
let index = parseInt(Math.random()*node.children.length,10);
transmit_child(node.children[index], event, _loop);
}
}
}
}
let nodes = document.querySelectorAll("[sec_auto_dom2_event_flag]");
if (nodes.length > 200) {
nodes = nodes.slice(0, 200);
}
nodes = window.randArr(nodes);
for (let node of nodes) {
let loop = 0;
await window.sleep(%f);
let event_name_list = node.getAttribute("sec_auto_dom2_event_flag").split("|");
let event_name_set = new Set(event_name_list);
event_name_list = [...event_name_set];
for (let event_name of event_name_list) {
let evt = document.createEvent('CustomEvent');
evt.initCustomEvent(event_name, true, true, null);
if (event_name == "click" || event_name == "focus" || event_name == "mouseover" || event_name == "select") {
transmit_child(node, evt, loop);
}
if ( (node.className && node.className.includes("close")) || (node.id && node.id.includes("close"))) {
continue;
}
try {
node.dispatchEvent(evt);
} catch(e) {}
}
}
})()
4.監控插入的節點，如果新增節點的href src含有JavaScript協議，則手動觸發。這似乎會漏一些內聯事件的觸發。(function init_observer_sec_auto_b() {
window.dom_listener_func_sec_auto = function (e) {
let node = e.target;
let nodeListSrc = node.querySelectorAll("[src]");
for (let each of nodeListSrc) {
if (each.src) {
window.addLink(each.src, "DOM");
let attrValue = each.getAttribute("src");
if (attrValue.toLocaleLowerCase().startsWith("javascript:")) {
try {
eval(attrValue.substring(11));
}
catch {}
}
}
}
let nodeListHref = node.querySelectorAll("[href]");
nodeListHref = window.randArr(nodeListHref);
for (let each of nodeListHref) {
if (each.href) {
window.addLink(each.href, "DOM");
let attrValue = each.getAttribute("href");
if (attrValue.toLocaleLowerCase().startsWith("javascript:")) {
try {
eval(attrValue.substring(11));
}
catch {}
}
}
}
};
document.addEventListener('DOMNodeInserted', window.dom_listener_func_sec_auto, true);
document.addEventListener('DOMSubtreeModified', window.dom_listener_func_sec_auto, true);
document.addEventListener('DOMNodeInsertedIntoDocument', window.dom_listener_func_sec_auto, true);
document.addEventListener('DOMAttrModified', window.dom_listener_func_sec_auto, true);
})()
窗口阻塞處理
crawler處理了 alert()/prompt() 基礎認證等等的阻塞。
chromedp.ListenTarget(*tab.Ctx, func(v interface{}) {
switch v := v.(type) {
//case *network.EventLoadingFailed:
// logger.Logger.Error("EventLoadingFailed ", v.ErrorText)
// 401 407 要求認證 此時會阻塞當前頁面 需要處理解決
case *fetch.EventAuthRequired:
tab.WG.Add(1)
go tab.HandleAuthRequired(v)
// close Dialog
case *page.EventJavascriptDialogOpening:
tab.WG.Add(1)
go tab.dismissDialog()
}
})
但是還有 打印 和 文件上傳窗口可能阻塞窗口
打印事件可以hook函數，文件上傳窗口可以用Page.setInterceptFileChooserDialog過濾。
End
一些還可以優化的部分，表單填充可以識別參數長度max-length、min-length
從Crawlergo的設計和源碼中能提取出很多東西來,
?基于網頁結構的大量網頁快速相似匹配，如果能集成到那些網絡空間引擎中應該會很好玩，但似乎還沒有一家做過。
有了原生的動態爬蟲支持，對自動化漏掃也有了更多的想法，例如通過hook一些觸發函數，污點檢測來檢測dom xss，爬蟲的原始請求包可以直接推到w13scan中。有了自動化爬蟲，后續所有流量都可以存儲一份，直接用搜索語法來找到相同參數的頁面進行poc測試等等。。
作者的代碼風格太不go了，想重寫一份了。
參考
?https://www.anquanke.com/post/id/178339#h2-17
?https://xz.aliyun.com/t/7064#toc-11
廣告
后續會在知識星球開源相關部分代碼，歡迎加入我的知識星球。
VSole
網絡安全專家