人需求真是复杂。又想要名字信息，又想要泛化的访问接口
反射实现的几种方案
用途，提供最基本的反射能力，即不需要指定的访问字段
这种设计即能保证tuple类型访问又能保留名字信息，通过静态反射来搞定
局限：仅仅支持简单的聚合类型(aggregate types)，多了继承就不行了,空基类也不行
struct simple_aggregate { // SimpleAggregare
std::string name;
int age;
boost::uuids::uuid uuid;
};
struct empty { // SimpleAggregare
};
struct aggregate : empty { // not a SimpleAggregare
std::string name;
int age;
boost::uuids::uuid uuid;
};
用法
#include <iostream>
#include <boost/pfr.hpp>
struct Record
{
std::string name;
int age;
double salary;
};
struct Point
{
int x;
int y;
};
int main()
{
Point pt{2, 3};
Record rec {"Baggins", 111, 999.99};
auto print = [](auto const& member) {
std::cout << member << " ";
};
boost::pfr::for_each_field(rec, print);
boost::pfr::for_each_field(pt, print);
}
文档里也介绍了原理
compile-time: use aggregate initialization to detect fields count in user-provided structure
BOOST_PFR_USE_CPP17 == 1:
BOOST_PFR_USE_CPP17 == 0 && BOOST_PFR_USE_LOOPHOLE == 1:
BOOST_PFR_USE_CPP17 == 0 && BOOST_PFR_USE_LOOPHOLE == 0:
现在是c++17～c++20了，考虑BOOST_PFR_USE_CPP17 == 1 就是利用结构化绑定和展开
原型大概这样
template <typename T, typename F>
// requires std::is_aggregate_v<T>
void for_each_member(T const & v, F f);
首先，我们要能探测出这个结构体有多少个字段
template <typename T>
constexpr auto size_()
-> decltype(T{\
{}, {}, {}, {}\
}, 0u)
{ return 4u; }
template <typename T>
constexpr auto size_()
-> decltype(T{\
{}, {}, {}\
}, 0u)
{ return 3u; }
template <typename T>
constexpr auto size_()
-> decltype(T{\
{}, {}\
}, 0u)
{ return 2u; }
template <typename T>
constexpr auto size_()
-> decltype(T{\
{}\
}, 0u)
{ return 1u; }
template <typename T>
constexpr auto size_()
-> decltype(T{}, 0u)
{ return 0u; }
template <typename T>
constexpr size_t size()
{
static_assert(std::is_aggregate_v<T>);
return size_<T>();
}
这段代码有点鬼畜， jeklly对两个括号没法解析，所以我加了斜杠
主要看这个decltype(T{\{}, {}\}, 0u), 要明白这是逗号表达式，左边的值是无所谓的，也就是说最后推导的肯定是usigned
但是能用T里面构造出来，就说明有几个字段，就匹配到了某个函数，返回值就是字段的个数
这里我们假定都是能用值初始化的，但可能某些字段不可以这样初始化，所以要加一个cast函数，来强制转换一下
struct init
{
template <typename T>
operator T(); // never defined
};
template <typename T>
constexpr auto size_()
-> decltype(T{init{}, init{}, init{}, init{}\
}, 0u)
{ return 4u; }
template <typename T>
constexpr auto size_()
-> decltype(T{init{}, init{}, init{}\
}, 0u)
{ return 3u; }
template <typename T>
constexpr auto size_()
-> decltype(T{init{}, init{}}, 0u)
{ return 2u; }
template <typename T>
constexpr auto size_()
-> decltype(T{init{}}, 0u)
{ return 1u; }
template <typename T>
constexpr auto size_()
-> decltype(T{}, 0u)
{ return 0u; }
template <typename T>
constexpr size_t size()
{
static_assert(std::is_aggregate_v<T>);
return size_<T>();
}
看上去可以了，但是size<Point>();还是会报错,因为简单类型不一定需要多个字段都初始化，所以可能会匹配多个
引入tag dispatch
template <unsigned I>
struct tag : tag<I - 1> {};
template <>
struct tag<0> {};
template <typename T>
constexpr auto size_(tag<4>)
-> decltype(T{init{}, init{}, init{}, init{}}, 0u)
{ return 4u; }
template <typename T>
constexpr auto size_(tag<3>)
-> decltype(T{init{}, init{}, init{}}, 0u)
{ return 3u; }
template <typename T>
constexpr auto size_(tag<2>)
-> decltype(T{init{}, init{}}, 0u)
{ return 2u; }
template <typename T>
constexpr auto size_(tag<1>)
-> decltype(T{init{}}, 0u)
{ return 1u; }
template <typename T>
constexpr auto size_(tag<0>)
-> decltype(T{}, 0u)
{ return 0u; }
template <typename T>
constexpr size_t size()
{
static_assert(std::is_aggregate_v<T>);
return size_<T>(tag<4>{}); // highest supported number
}
这样就不会匹配错误了
对应的for_each就是结构化绑定
template <typename T, typename F>
void for_each_member(T const& v, F f)
{
static_assert(std::is_aggregate_v<T>);
if constexpr (size<T>() == 4u)
{
const auto& [m0, m1, m2, m3] = v;
f(m0); f(m1); f(m2); f(m3);
}
else if constexpr (size<T>() == 3u)
{
const auto& [m0, m1, m2] = v;
f(m0); f(m1); f(m2);
}
else if constexpr (size<T>() == 2u)
{
const auto& [m0, m1] = v;
f(m0); f(m1);
}
else if constexpr (size<T>() == 1u)
{
const auto& [m0] = v;
f(m0);
}
}
知道size就好泛化了。
boost.pfr实现的更加泛化，有机会可以研究研究
这个函数没啥好说的，主要是为了偷东西 诞生的,实现非常简单
template<class T, class U = T>
constexpr // since C++20
T exchange(T& obj, U&& new_value)
{
T old_value = std::move(obj);
obj = std::forward<U>(new_value);
return old_value;
}
比如参考链接1里面 move constructor的实现
struct S
{
int n;
S(S&& other) noexcept : n{std::exchange(other.n, 0)}
{}
S& operator=(S&& other) noexcept
{
if(this != &other)
n = std::exchange(other.n, 0); // 移动 n ，并于 other.n 留下零
return *this;
}
};
我看到的用法
template <promise_base::urgent Urgent>
void promise_base::make_ready() noexcept {
if (_task) {
if (Urgent == urgent::yes) {
::seastar::schedule_urgent(std::exchange(_task, nullptr));
} else {
::seastar::schedule(std::exchange(_task, nullptr));
}
}
}
可能就要比较std::swap和他的区别了，直接上结论吧，上限是std::swap的性能，要不是为了偷东西这个特性，不要用
SO有个链接做了简单验证，见参考链接2
然后Ben Deane 有个案例 std::exchange 惯用法，在参考链接3 4 里。简单概括下
就是用std:exchange 来省掉没必要的临时变量，链接3 的ppt可以看下，写的很漂亮，作者叫他 The “swap-and-iterate” pattern
我把参考链接四的代码贴一下
以前用swap
class Dispatcher {
// We hold some vector of callables that represents
// events to dispatch or actions to take
using Callback = /* some callable */;
std::vector<Callback> callbacks_;
// Anyone can register an event to be dispatched later
void defer_event(const Callback& cb) {
callbacks_.push_back(cb);
}
// All events are dispatched when we call process
void process() {
std::vector<Callback> tmp{};
using std::swap; // the "std::swap" two-step
swap(tmp, callbacks_);
for (const auto& callback : tmp) {
std::invoke(callback);
}
}
void post_event(Callback& cb) {
Callback tmp{};
using std::swap;
swap(cb, tmp);
PostToMainThread([this, cb_ = std::move(tmp)] {
callbacks_.push_back(cb_);
});
}
};
改成exchange
class Dispatcher {
// ...
// All events are dispatched when we call process
void process() {
for (const auto& callback : std::exchange(callbacks_, {}) {
std::invoke(callback);
}
}
void post_event(Callback& cb) {
PostToMainThread([this, cb_ = std::exchange(cb, {})] {
callbacks_.push_back(cb_);
});
}
};
可能你会问，直接std::move不就完事儿，这里作者强调接口的灵活性？
强调move并不会empty,并不会clear，可能还有值，比如std::optional
结合lock
原本std::swap 是这样的
class Dispatcher {
// ...
// All events are dispatched when we call process
void process() {
std::vector<Callback> tmp{};
{
using std::swap;
std::scoped_lock lock{mutex_};
swap(tmp, callbacks_);
}
for (const auto& callback : tmp) {
std::invoke(callback);
}
}
};
改成exchange 省掉一个数组
class Dispatcher {
// ...
// All events are dispatched when we call process
void process() {
std::scoped_lock lock{mutex_};
for (const auto& callback : std::exchange(callbacks_, {})) {
std::invoke(callback);
}
}
};
能不能吧lock也省掉？临时变量声明周期是一行，一行就够了
class Dispatcher {
// ...
// All events are dispatched when we call process
void process() {
const auto tmp = (std::scoped_lock{mutex_}, std::exchange(callbacks_, {}));
for (const auto& callback : tmp) {
std::invoke(callback);
}
}
};
文章摘抄自这里
场景，当访问不合法的地址，当场segment fault，为了避免，如何探测？
两种方案
作者写了个简单的代码，这里直接列出来看看原理即可
#define _GNU_SOURCE
#include <stdint.h>
#include <signal.h>
#include <assert.h>
#include <stdlib.h>
#include <stdio.h>
#include <ucontext.h>
#ifdef __i386__
typedef uint32_t word_t;
#define IP_REG REG_EIP
#define IP_REG_SKIP 3
#define READ_CODE __asm__ __volatile__(".byte 0x8b, 0x03\n" /* mov (%ebx), %eax */ \
".byte 0x41\n" /* inc %ecx */ \
: "=a"(ret), "=c"(tmp) : "b"(addr), "c"(tmp));
#endif
#ifdef __x86_64__
typedef uint64_t word_t;
#define IP_REG REG_RIP
#define IP_REG_SKIP 6
#define READ_CODE __asm__ __volatile__(".byte 0x48, 0x8b, 0x03\n" /* mov (%rbx), %rax */ \
".byte 0x48, 0xff, 0xc1\n" /* inc %rcx */ \
: "=a"(ret), "=c"(tmp) : "b"(addr), "c"(tmp));
#endif
static void segv_action(int sig, siginfo_t *info, void *ucontext) {
(void) sig;
(void) info;
ucontext_t *uctx = (ucontext_t*) ucontext;
uctx->uc_mcontext.gregs[IP_REG] += IP_REG_SKIP;
}
struct sigaction peek_sigaction = {
.sa_sigaction = segv_action,
.sa_flags = SA_SIGINFO,
.sa_mask = 0,
};
word_t peek(word_t *addr, int *success) {
word_t ret;
int tmp, res;
struct sigaction prev_act;
res = sigaction(SIGSEGV, &peek_sigaction, &prev_act);
assert(res == 0);
tmp = 0;
READ_CODE
res = sigaction(SIGSEGV, &prev_act, NULL);
assert(res == 0);
if (success) {
*success = tmp;
}
return ret;
}
int main() {
int success;
word_t number = 22;
word_t value;
number = 22;
value = peek(&number, &success);
printf("%d %d\n", success, value);
value = peek(NULL, &success);
printf("%d %d\n", success, value);
value = peek((word_t*)0x1234, &success);
printf("%d %d\n", success, value);
return 0;
}
看一乐啊，这里就是操作指针对应的寄存器，不保证正确（多线程下应该不对，这东西应该说进程级别的，放在最外层）
另外，如果不是写什么共享内存程序，segment fault 就挂了得了，别挽救了
Practical memory pool based allocators for Modern C++
又讲内存池实现的，内存池等于块池
bucket为基本单元 bucket has two properties: BlockSize and BlockCount
bucket主要接口，构造析构分配回收
class bucket {
public:
const std::size_t BlockSize;
const std::size_t BlockCount;
bucket(std::size_t block_size, std::size_t block_count);
~bucket();
// Tests if the pointer belongs to this bucket
bool belongs(void * ptr) const noexcept;
// Returns nullptr if failed
[[nodiscard]] void * allocate(std::size_t bytes) noexcept;
void deallocate(void * ptr, std::size_t bytes) noexcept;
private:
// Finds n free contiguous blocks in the ledger and returns the first block’s index or BlockCount on failure
std::size_t find_contiguous_blocks(std::size_t n) const noexcept;
// Marks n blocks in the ledger as “in-use” starting at ‘index’
void set_blocks_in_use(std::size_t index, std::size_t n) noexcept;
// Marks n blocks in the ledger as “free” starting at ‘index’
void set_blocks_free(std::size_t index, std::size_t n) noexcept;
// Actual memory for allocations
std::byte* m_data{nullptr};
// Reserves one bit per block to indicate whether it is in-use
std::byte* m_ledger{nullptr};
};
bucket::bucket(std::size_t block_size, std::size_t block_count)
: BlockSize{block_size}
, BlockCount{block_count}
{
const auto data_size = BlockSize * BlockCount;
m_data = static_cast<std::byte*>(std::malloc(data_size));
assert(m_data != nullptr);
const auto ledger_size = 1 + ((BlockCount - 1) / 8);
m_ledger = static_cast<std::byte*>(std::malloc(ledger_size));
assert(m_ledger != nullptr);
std::memset(m_data, 0, data_size);
std::memset(m_ledger, 0, ledger_size);
}
bucket::~bucket() {
std::free(m_ledger);
std::free(m_data);
}
void * bucket::allocate(std::size_t bytes) noexcept {
// Calculate the required number of blocks
const auto n = 1 + ((bytes - 1) / BlockSize);
const auto index = find_contiguous_blocks(n);
if (index == BlockCount) {
return nullptr;
}
set_blocks_in_use(index, n);
return m_data + (index * BlockSize);
}
void bucket::deallocate(void * ptr, std::size_t bytes) noexcept {
const auto p = static_cast<const std::byte *>(ptr);
const std::size_t dist = static_cast<std::size_t>(p - m_data);
// Calculate block index from pointer distance
const auto index = dist / BlockSize;
// Calculate the required number of blocks
const auto n = 1 + ((bytes - 1) / BlockSize);
// Update the ledger
set_blocks_free(index, n);
}
然后就是由块来构成池子 指定 BlockSize和BlockCount
// The default implementation defines a pool with no buckets
template<std::size_t id>
struct bucket_descriptors {
using type = std::tuple<>;
};
struct bucket_cfg16 {
static constexpr std::size_t BlockSize = 16;
static constexpr std::size_t BlockCount = 10000;
};
struct bucket_cfg32{
static constexpr std::size_t BlockSize = 32;
static constexpr std::size_t BlockCount = 10000;
};
struct bucket_cfg1024 {
static constexpr std::size_t BlockSize = 1024;
static constexpr std::size_t BlockCount = 50000;
};
template<>
struct bucket_descriptors<1> {
using type = std::tuple<bucket_cfg16, bucket_cfg32, bucket_cfg1024>;
};
template<std::size_t id>
using bucket_descriptors_t = typename bucket_descriptors<id>::type;
template<std::size_t id>
static constexpr std::size_t bucket_count = std::tuple_size<bucket_descriptors_t<id>>::value;
template<std::size_t id>
using pool_type = std::array<bucket, bucket_count<id>>;
template<std::size_t id, std::size_t Idx>
struct get_size
: std::integral_constant<std::size_t, std::tuple_element_t<Idx, bucket_descriptors_t<id>>::BlockSize>{\\
};
template<std::size_t id, std::size_t Idx>
struct get_count
: std::integral_constant<std::size_t, std::tuple_element_t<Idx, bucket_descriptors_t<id>>::BlockCount>{\\
};
template<std::size_t id, std::size_t... Idx>
auto & get_instance(std::index_sequence<Idx...>) noexcept {
static pool_type<id> instance{\{\{get_size<id, Idx>::value, get_count<id, Idx>::value} ...\}\};
return instance;
}
template<std::size_t id>
auto & get_instance() noexcept {
return get_instance<id>(std::make_index_sequence<bucket_count<id>>());
}
涉及到具体的分配策略，怎么找到所需的块呢？
直接找空闲的 有点像hash_map 开放地址法实现。浪费
// Assuming buckets are sorted by their block sizes
template<std::size_t id>
[[nodiscard]] void * allocate(std::size_t bytes) {
auto & pool = get_instance<id>();
for (auto & bucket : pool) {
if(bucket.BlockSize >= bytes) {
if(auto ptr = bucket.allocate(bytes); ptr != nullptr) {
return ptr;
}
}
}
throw std::bad_alloc{};
}
需要额外的信息
template<std::size_t id>
[[nodiscard]] void * allocate(std::size_t bytes) {
auto & pool = get_instance<id>();
std::array<info, bucket_count<id>> deltas;
std::size_t index = 0;
for (const auto & bucket : pool) {
deltas[index].index = index;
if (bucket.BlockSize >= bytes) {
deltas[index].waste = bucket.BlockSize - bytes;
deltas[index].block_count = 1;
} else {
const auto n = 1 + ((bytes - 1) / bucket.BlockSize);
const auto storage_required = n * bucket.BlockSize;
deltas[index].waste = storage_required - bytes;
deltas[index].block_count = n;
}
++index;
}
sort(deltas.begin(), deltas.end()); // std::sort() is allowed to allocate
for (const auto & d : deltas)
if (auto ptr = pool[d.index].allocate(bytes); ptr != nullptr)
return ptr;
throw std::bad_alloc{};
}
碎片问题？
实现allocator接口
不讲了。看代码
template<typename T = std::uint8_t, std::size_t id = 0>
class static_pool_allocator {
public:
//rebind不用实现吧，我记得好像废弃了
template<typename U>
static_pool_allocator(const static_pool_allocator<U, id> & other) noexcept
: m_upstream_resource{other.upstream_resource()} {}
template<typename U>
static_pool_allocator & operator=(const static_pool_allocator<U, id> & other) noexcept {
m_upstream_resource = other.upstream_resource();
return *this;
}
static bool initialize_memory_pool() noexcept { return memory_pool::initialize<id>(); }
private:
pmr::memory_resource * m_upstream_resource;
};
后面介绍了个分析allocate的工具
clang
pass长这样
class AllocListPass : public llvm::FunctionPass {
public:
static char ID;
AllocListPass() : llvm::FunctionPass(ID) {}
bool runOnFunction(llvm::Function & f) override {
const auto pretty_name = boost::core::demangle(f.getName().str().c_str());
static const std::regex call_regex{R"(void instrument::type_reg<([^,]+),(.+),([^,]+)>\(\))"};
std::smatch match;
if (std::regex_match(pretty_name, match, call_regex)) {
if (match.size() == 4) {
const auto pool_id = std::atoi(match[1].str().c_str());
const auto type = match[2].str();
const auto size = std::atoi(match[3].str().c_str());
std::cout << "Pool ID: " << pool_id << ", Size: " << size << ", Type: " << type << "\n";
}
}
return false; // does not alter the code, a read-only pass
}
};
char AllocListPass::ID = 0;
static llvm::RegisterPass<AllocListPass> dummy("alloc-list", "This pass lists memory pool allocations");
llvm::ModulePass原理
dfs找入口 main等
结果这样
Call graph for: Pool ID: 3, Size: 24, Type: std::__1::__list_node<int, void*>:
1. static_pool_allocator<std::__1::__list_node<int, void*>, 3ul>::allocate(unsigned long, void const*) called at /usr/include/c++/v1/memory:1547
2. std::__1::allocator_traits<static_pool_allocator<std::__1::__list_node<int, void*>, 3ul>>::allocate(static_pool_allocator<std::__1::__list_node<int, void*>, 3ul>&,
unsigned long) called at /usr/include/c++/v1/list:1079
3. std::__1::list<int, static_pool_allocator<int, 3ul>>::__allocate_node(static_pool_allocator<std::__1::__list_node<int, void*>, 3ul>&) called at
/usr/include/c++/v1/list:1569
4. std::__1::list<int, static_pool_allocator<int, 3ul>>::push_back(int const&) called at /home/program.cpp:12
5. x() called at /home/program.cpp:7
6. f() called at /home/program.cpp:2
llvm opt
opt -load alloc-analyzer.so -alloc-analyze -gen-hdr my_defs.hpp -entry-point "main"< home/program.bc -o /dev/null
指 pair tuple (product type)(结构体) optional variant(sum type) (有index信息)
std::any不行，有类型擦除，丢信息了
pair tuple 多种信息
optional，优雅，不浪费堆空间
optional 默认值, 没有值 对于指针场景，没有堆使用。优雅
std::unique_ptr<ComplicatedObject> obj_ = nullptr;
void setComplicated(int a, int b) {
obj_ = std::make_unique<ComplicatedObject>(a, b);
}
std::optional<ComplicatedObject> obj_ = std::nullopt;
void setComplicated(int a, int b) {
obj_.emplace(a, b);
}
variant 优雅的union
index方法 返回下表
std::get可以 通过index和类型来访问 类似的std::get_if
std::visit
poor man’s Expected<T>.
std::variant<std::string, std::errc> vGetenv(const char *name);
if (auto v = vGetenv("foo"); std::get_if<std::string>(&v)) {
const auto& value = std::get<std::string>(v);
std::cout << "Value is: " << value << "\n";
} else {
std::error_condition error = std::get<std::errc>(v);
std::cout << "Error was: " << error.message() << "\n";
}
静态变量，非虚成员函数，静态成员函数，类型成员不会影响类的存储布局
空基类，[[no_unique_address]]修饰，强制指定不占用空间
比较，auto operator <=>(const Flatland &) const = default;
POD，以及pack
class NarrowLand {
unsigned char x; // offset 0
unsigned long long y; // offset 8 (still!)
unsigned long long z; // offset 16
friend bool operator ==(NarrowLand const &lhs, NarrowLand const &rhs);
};
bool operator ==(NarrowLand const &lhs, NarrowLand const &rhs) {
if constexpr (has_unique_object_representations_v<NarrowLand>)
return !memcmp(&lhs, &rhs, sizeof(NarrowLand));
else
return lhs.x == rhs.x && lhs.y == rhs.y && lhs.z == rhs.z;
}
vptr
介绍了一波实现。太累了。不看了。这段东西看着就头疼
What is a data race and how do we fix it?
struct TokenPool {
std::vector<Token> tokens_;
std::mutex mtx_;
std::condition_variable cv_;
void returnToken(Token t) {
std::unique_lock lk(mtx_);
tokens_.push_back(t);
lk.unlock();//!
cv_.notify_one();
}
Token getToken() {
std::unique_lock lk(mtx_);
while (tokens_.empty()) {
cv_.wait(lk);
}
Token t = std::move(tokens_.back());
tokens_.pop_back();
return t;
}
};
Static initialization and once_flag 多线程的初始化
class Logger {
std::mutex mtx_;
std::optional<NetworkConnection> conn_;
NetworkConnection& getConn() {
std::lock_guard<std::mutex> lk(mtx_);
if (!conn_.has_value()) {
conn_ = NetworkConnection(defaultHost);
}
return *conn_;
}
};
class Logger {
std::once_flag once_;
std::optional<NetworkConnection> conn_;
NetworkConnection& getConn() {
std::call_once(once_, []() {
conn_ = NetworkConnection(defaultHost);
});
return *conn_;
}
};
mutex |
condition_variable |
once_flag |
|---|---|---|
lock blocks only if someone “owns” the mutex. |
wait always blocks. |
call_once blocks only if the “done” flag isn’t yet set. |
Many threads can queue up on lock. |
Many threads can queue up on wait. |
Many threads can queue up on call_once. |
Calling unlock unblocks exactly one waiter: the new “owner.” |
Calling notify_one unblocks exactly one waiter. |
Failing at the callback unblocks exactly one waiter: the new “owner.” |
Calling notify_all unblocks all waiters. |
Succeeding at the callback unblocks all waiters and sets the “done” flag. |
New C++17 and C++20 primitives
using Sem = std::counting_semaphore<256>;
struct SemReleaser {
bool operator()(Sem *s) const { s->release(); }
};
class AnonymousTokenPool {
Sem sem_{100};
using Token = std::unique_ptr<Sem, SemReleaser>;
Token borrowToken() {
sem_.acquire(); // may block
return Token(&sem_);
}
};
Patterns for sharing data
Remember: Producer/consumer? Use mutex + condition_variable.
In conclusion
mutex + condition_variable are best friends
C++20 gives us “counting” primitives like semaphore and latch
But if your program is fundamentally multithreaded, look for higher-level facilities: promise/future, coroutines, ASIO, TBB
noexceptc++20
[capture clause] <template parameters\> (parameter list)
specifier exception attribute -> return type requires { body }
std::unique_ptr<Widget> myPtr = std::make_unique<Widget>();
auto myLamb = [ capturedPtr = std::move(myPtr) ] ( )
{ return capturedPtr->computeSize(); };
再谈右值
int&& func() { return 42; }
void test() {
int a = func();//返回之前已经销毁
}
std::move
template <class T>
constexpr remove_reference_t<T>&& move(T&& t) noexcept
{
return static_cast<remove_reference_t<T>&&>(t);
}
move ctor
struct S {
double* data;
S( S&& other ) noexcept
: data(std::exchange(other.data, nullptr))
{ }
S& operator=( S&& other ) noexcept {
if (this == &other) return *this;
delete[] data;
data = std::exchange(other.data, nullptr);
return *this;
}
};
完美转发
template <class T> void f(T&& value)
{
g(std::forward<T>(value));
}
有些类型只能move不能copy
使用建议，最好别用share or
c++17引入CTAD 没啥说的
using
template<size_t N>
using CharArray = std::array<char, N>;
std::array受限于NTTP这种参数难受，不如std::span
变参模板
SFINAE
讲了一遍编译原理
ODR
ABI
name-mangling
变量存储在哪
1.66版本，boost.asio库重新设计了框架，目前最新版为1.71。读了几天代码后，对框架中相关概念总结。因为是泛型编程的库，所以分析的概念层的设计。
可通过boost官方文档，strand的1.65和1.66两版本文档比较，查证ts和io_context, executor首次出现在1.66。
新框架有几个核心概念，Context，Scheduler，Service，Executor，Strand。
Context:
Scheduler:
Service:
Executor:
Strand:
环境
go version #go version go1.10.4 linux/amd64
lsb_release -d #Description: Ubuntu 18.04.1 LTS
gdb --version #GNU gdb (Ubuntu 8.1-0ubuntu3.2) 8.1.0.20180409-git
测试代码 test.go
package main
func main() {
println("hello, world");
}
go build -gcflags "-N -l" -o test test.go
gdb test
(gdb) info files
Symbols from "/mnt/c/Program Files/cmder/test".
Local exec file:
`/mnt/c/Program Files/cmder/test', file type elf64-x86-64.
Entry point: 0x4477c0
0x0000000000401000 - 0x000000000044c213 is .text
0x000000000044d000 - 0x00000000004757a3 is .rodata
0x00000000004758e0 - 0x0000000000475f80 is .typelink
0x0000000000475f80 - 0x0000000000475f88 is .itablink
0x0000000000475f88 - 0x0000000000475f88 is .gosymtab
0x0000000000475fa0 - 0x00000000004a3630 is .gopclntab
0x00000000004a4000 - 0x00000000004a4a08 is .noptrdata
0x00000000004a4a20 - 0x00000000004a65b0 is .data
0x00000000004a65c0 - 0x00000000004c2888 is .bss
0x00000000004c28a0 - 0x00000000004c4e58 is .noptrbss
0x0000000000400f9c - 0x0000000000401000 is .note.go.buildid
(gdb) b *0x4477c0
Breakpoint 1 at 0x4477c0: file /usr/lib/go-1.10/src/runtime/rt0_linux_amd64.s, line 8.
版本对应的汇编有变化，没有明显的main，但是入口肯定是_rt0_amd64
#include "textflag.h"
TEXT _rt0_amd64_linux(SB),NOSPLIT,$-8
JMP _rt0_amd64(SB)
TEXT _rt0_amd64_linux_lib(SB),NOSPLIT,$0
JMP _rt0_amd64_lib(SB)
(gdb) b _rt0_amd64
Breakpoint 2 at 0x444100: file /usr/lib/go-1.10/src/runtime/asm_amd64.s, line 15.
对应汇编是书里的runtime.rt0_go
TEXT _rt0_amd64(SB),NOSPLIT,$-8
MOVQ 0(SP), DI // argc
LEAQ 8(SP), SI // argv
JMP runtime·rt0_go(SB)
b runtime.rt0_go
Breakpoint 3 at 0x444110: file /usr/lib/go-1.10/src/runtime/asm_amd64.s, line 89.
;前面有很多对于汇编指令cpu类型的判断，参数入栈等等
// create a new goroutine to start program
MOVQ $runtime·mainPC(SB), AX // entry
PUSHQ AX
PUSHQ $0 // arg size
CALL runtime·newproc(SB)
POPQ AX
POPQ AX
// start this M
CALL runtime·mstart(SB)
MOVL $0xf1, 0xf1 // crash
RET
DATA runtime·mainPC+0(SB)/8,$runtime·main(SB)
GLOBL runtime·mainPC(SB),RODATA,$8
b runtime.schedinit
Breakpoint 6 at 0x423a60: file /usr/lib/go-1.10/src/runtime/proc.go, line 477.
b runtime.main
Breakpoint 4 at 0x4228b0: file /usr/lib/go-1.10/src/runtime/proc.go, line 109.
schedinit 入口
// The bootstrap sequence is:
//
// call osinit
// call schedinit
// make & queue new G
// call runtime·mstart
//
// The new G calls runtime·main.
func schedinit() {
// raceinit must be the first call to race detector.
// In particular, it must be done before mallocinit below calls racemapshadow.
_g_ := getg()
if raceenabled {
_g_.racectx, raceprocctx0 = raceinit()
}
sched.maxmcount = 10000
tracebackinit()
moduledataverify()
stackinit()
mallocinit()
mcommoninit(_g_.m)
alginit() // maps must not be used before this call
modulesinit() // provides activeModules
typelinksinit() // uses maps, activeModules
itabsinit() // uses activeModules
msigsave(_g_.m)
initSigmask = _g_.m.sigmask
goargs()
goenvs()
//处理GODEBUG GOTRACEBACK宏
parsedebugvars()
//垃圾回收器初始化
gcinit()
sched.lastpoll = uint64(nanotime())
//通过CPU core和GOMAXPROCS确定P数量
procs := ncpu
if n, ok := atoi32(gogetenv("GOMAXPROCS")); ok && n > 0 {
procs = n
}
// 调整P数量
if procresize(procs) != nil {
throw("unknown runnable goroutine during bootstrap")
}
// For cgocheck > 1, we turn on the write barrier at all times
// and check all pointer writes. We can't do this until after
// procresize because the write barrier needs a P.
if debug.cgocheck > 1 {
writeBarrier.cgo = true
writeBarrier.enabled = true
for _, p := range allp {
p.wbBuf.reset()
}
}
下一步是runtime.main
// The main goroutine.
func main() {
g := getg()
// Racectx of m0->g0 is used only as the parent of the main goroutine.
// It must not be used for anything else.
g.m.g0.racectx = 0
// Max stack size is 1 GB on 64-bit, 250 MB on 32-bit.
// Using decimal instead of binary GB and MB because
// they look nicer in the stack overflow failure message.
if sys.PtrSize == 8 {
maxstacksize = 1000000000
} else {
maxstacksize = 250000000
}
// Allow newproc to start new Ms.
//启动系统后台监控/定期垃圾回收，并发任务调度相关
mainStarted = true
systemstack(func() {
newm(sysmon, nil)
})
// Lock the main goroutine onto this, the main OS thread,
// during initialization. Most programs won't care, but a few
// do require certain calls to be made by the main thread.
// Those can arrange for main.main to run in the main thread
// by calling runtime.LockOSThread during initialization
// to preserve the lock.
lockOSThread()
if g.m != &m0 {
throw("runtime.main not on m0")
}
runtime_init() // must be before defer
if nanotime() == 0 {
throw("nanotime returning zero")
}
// Defer unlock so that runtime.Goexit during init does the unlock too.
needUnlock := true
defer func() {
if needUnlock {
unlockOSThread()
}
}()
// Record when the world started. Must be after runtime_init
// because nanotime on some platforms depends on startNano.
runtimeInitTime = nanotime()
gcenable()
main_init_done = make(chan bool)
if iscgo {
if _cgo_thread_start == nil {
throw("_cgo_thread_start missing")
}
if GOOS != "windows" {
if _cgo_setenv == nil {
throw("_cgo_setenv missing")
}
if _cgo_unsetenv == nil {
throw("_cgo_unsetenv missing")
}
}
if _cgo_notify_runtime_init_done == nil {
throw("_cgo_notify_runtime_init_done missing")
}
// Start the template thread in case we enter Go from
// a C-created thread and need to create a new thread.
startTemplateThread()
cgocall(_cgo_notify_runtime_init_done, nil)
}
fn := main_init // make an indirect call, as the linker doesn't know the address of the main package when laying down the runtime
fn()
close(main_init_done)
needUnlock = false
unlockOSThread()
if isarchive || islibrary {
// A program compiled with -buildmode=c-archive or c-shared
// has a main, but it is not executed.
return
}
fn = main_main // make an indirect call, as the linker doesn't know the address of the main package when laying down the runtime
fn()
if raceenabled {
racefini()
}
// Make racy client program work: if panicking on
// another goroutine at the same time as main returns,
// let the other goroutine finish printing the panic trace.
// Once it does, it will exit. See issues 3934 and 20018.
if atomic.Load(&runningPanicDefers) != 0 {
// Running deferred functions should not take long.
for c := 0; c < 1000; c++ {
if atomic.Load(&runningPanicDefers) == 0 {
break
}
Gosched()
}
}
if atomic.Load(&panicking) != 0 {
gopark(nil, nil, "panicwait", traceEvGoStop, 1)
}
exit(0)
//? 这啥
for {
var x *int32
*x = 0
}
一个复杂示例
//cat lib/sum.go
package lib
func init() {
println("sum.init")
}
func Sum(x ...int) int {
n := 0
for _, i := range x{
n += i
}
return n
}
//cat test.go
package main
import (
"./lib"
)
func init() {
println("test.init")
}
func test() {
println(lib.Sum(1,2,3))
}
//cat main.go
package main
import (
_ "net/http"
)
func init() {
println("main.init.2")
}
func main() {
test()
}
func init() {
println("main.init.1")
}
执行结果
go build -gcflags "-N -l" -o test
./test
sum.init
main.init.2
main.init.1
test.init
6
查看反汇编
;go tool objdump -s "runtime\.init\b" test
TEXT runtime.init.0(SB) /usr/lib/go-1.10/src/runtime/cpuflags_amd64.go
TEXT runtime.init.1(SB) /usr/lib/go-1.10/src/runtime/mgcwork.go
mgcwork.go:25 0x420860 c3 RET
TEXT runtime.init.2(SB) /usr/lib/go-1.10/src/runtime/mstats.go
mstats.go:438 0x4260d0 64488b0c25f8ffffff MOVQ
TEXT runtime.init.3(SB) /usr/lib/go-1.10/src/runtime/panic.go
TEXT runtime.init.4(SB) /usr/lib/go-1.10/src/runtime/proc.go
TEXT runtime.init.5(SB) /usr/lib/go-1.10/src/runtime/signal_unix.go
signal_unix.go:64 0x43e450 c3 RET
TEXT runtime.init(SB) <autogenerated>
;go tool objdump -s "main\.init\b" test
TEXT main.init.0(SB) /mnt/c/Program Files/cmder/main.go
TEXT main.init.1(SB) /mnt/c/Program Files/cmder/main.go
TEXT main.init.2(SB) /mnt/c/Program Files/cmder/test.go
TEXT main.init(SB) <autogenerated>
<autogenerated>:1 0x5e31ec e81f63ffff CALL net/http.init(SB)
<autogenerated>:1 0x5e31f1 e83afdffff CALL _/mnt/c/Program_Files/cmder/lib.init(SB)
<autogenerated>:1 0x5e31f6 e895fdffff CALL main.init.0(SB)
<autogenerated>:1 0x5e31fb e820feffff CALL main.init.1(SB)
<autogenerated>:1 0x5e3200 e87bfeffff CALL main.init.2(SB)
<autogenerated>:1 0x5e3205 c605822a1f0002 MOVB $0x2, main.initdone.(SB)
<autogenerated>:1 0x5e320c 488b2c24 MOVQ 0(SP), BP
<autogenerated>:1 0x5e3210 4883c408 ADDQ $0x8, SP
<autogenerated>:1 0x5e3214 c3 RET
<autogenerated>:1 0x5e3215 e80600e7ff CALL runtime.morestack_noctxt(SB)
<autogenerated>:1 0x5e321a eb84 JMP main.init(SB)
结论
所有init都会在同一个goroutine执行
所有init函数结束后才会执行main.main
基本策略
内存块
初始化动作
三个数组组成内存管理结构
arena和spans bitmap存在映射关系，三者可以按需同步线性扩张
都用mheap维护，在mallocinit里初始化
来个示例
//test.go
package main
import(
"fmt"
"os"
"github.com/shirou/gosutil/process"
)
var ps *process.Process
func mem(n int) {
if ps == nil {
p, err := process.NewProcess(int32(os.Getpid()))
if err != nil {
panic(err)
}
ps = p
}
mem, _ := ps.MemoryInfoEx()
fmt.Printf("%d, VMS:%d MB, RSS:%d MB\n", n, mem,.VMS>>20, mem.RSS>>20)
}
func main(){
mem(1)
data : new([10][1024*1024]byte)
mem(2)
for i := range data {
for x, n := 0, len(data[i]); x<n; x++ {
data[i][x] = 1
}
mem(3)
}
}
分配
不要以为new一定会分配在堆上，随着优化内联
package main
import ()
func test() *int {
x := new(int)
*x = 0xAABB
return x
}
func main() {
println(*test())
}
go build -gcflags "-l" -o test test.go
go tool objdump -s "main\.test" test
TEXT main.test(SB) /mnt/c/Program Files/cmder/test.go
test.go:4 0x44c150 64488b0c25f8ffffff MOVQ FS:0xfffffff8, CX
test.go:4 0x44c159 483b6110 CMPQ 0x10(CX), SP
test.go:4 0x44c15d 7639 JBE 0x44c198
test.go:4 0x44c15f 4883ec18 SUBQ $0x18, SP
test.go:4 0x44c163 48896c2410 MOVQ BP, 0x10(SP)
test.go:4 0x44c168 488d6c2410 LEAQ 0x10(SP), BP
test.go:5 0x44c16d 488d05acac0000 LEAQ 0xacac(IP), AX
test.go:5 0x44c174 48890424 MOVQ AX, 0(SP)
test.go:5 0x44c178 e8a3effbff CALL runtime.newobject(SB)
test.go:5 0x44c17d 488b442408 MOVQ 0x8(SP), AX
test.go:6 0x44c182 48c700bbaa0000 MOVQ $0xaabb, 0(AX)
test.go:7 0x44c189 4889442420 MOVQ AX, 0x20(SP)
test.go:7 0x44c18e 488b6c2410 MOVQ 0x10(SP), BP
test.go:7 0x44c193 4883c418 ADDQ $0x18, SP
test.go:7 0x44c197 c3 RET
test.go:4 0x44c198 e8d383ffff CALL runtime.morestack_noctxt(SB)
test.go:4 0x44c19d ebb1 JMP main.test(SB)
go build -o test test.go
go tool objdump -s "main\.main" test
TEXT main.main(SB) /mnt/c/Program Files/cmder/test.go
test.go:10 0x44c150 64488b0c25f8ffffff MOVQ FS:0xfffffff8, CX
test.go:10 0x44c159 483b6110 CMPQ 0x10(CX), SP
test.go:10 0x44c15d 7634 JBE 0x44c193
test.go:10 0x44c15f 4883ec10 SUBQ $0x10, SP
test.go:10 0x44c163 48896c2408 MOVQ BP, 0x8(SP)
test.go:10 0x44c168 488d6c2408 LEAQ 0x8(SP), BP
test.go:11 0x44c16d e88e59fdff CALL runtime.printlock(SB)
test.go:11 0x44c172 48c70424bbaa0000 MOVQ $0xaabb, 0(SP)
test.go:11 0x44c17a e80161fdff CALL runtime.printint(SB)
test.go:11 0x44c17f e80c5cfdff CALL runtime.printnl(SB)
test.go:11 0x44c184 e8f759fdff CALL runtime.printunlock(SB)
test.go:12 0x44c189 488b6c2408 MOVQ 0x8(SP), BP
test.go:12 0x44c18e 4883c410 ADDQ $0x10, SP
test.go:12 0x44c192 c3 RET
test.go:10 0x44c193 e8d883ffff CALL runtime.morestack_noctxt(SB)
test.go:10 0x44c198 ebb6 JMP main.main(SB)
逃逸分析-gcflag “-m”
分配思路 malloc.go
回收
回收以span为单位
释放
sysmon监控任务来搞
具体释放是madvie(v, n, _MADV_DONTNEED) 系统来决定。如果物理内存资源充足，就不会回收避免无谓的损耗，不过再次使用肯定会pagefault然后分配新的内存
缩短STW时间
抑制堆增长 充分利用CPU资源
gcController控制
辅助回收，避免分配速度大于后台标记导致的堆恶性扩张
公司不让访问外网了呜呜
https://support.hypernode.com/en/troubleshooting/performance/how-to-debug-out-of-memory-oom-events
https://briancallahan.net/blog/20200816.html
https://lobste.rs/s/2vj4sb/file_handling_unix_tips_traps_outright
https://danluu.com/file-consistency/
https://rachelbythebay.com/w/2020/08/11/files/
https://robertovitillo.com/what-every-developer-should-know-about-database-consistency/
http://brooker.co.za/blog/2020/05/25/reading.html
https://codahale.com/work-is-work/
https://bartoszmilewski.com/2020/08/11/benign-data-races-considered-harmful/
https://www.fluentcpp.com/2020/06/12/a-generic-component-for-out-of-line-lambdas/
http://brooker.co.za/blog/2020/03/22/rust.html
https://briancallahan.net/blog/20200812.html
https://blog.kevinjahns.de/are-crdts-suitable-for-shared-editing/
https://zhuanlan.zhihu.com/p/108968057
https://www.cockroachlabs.com/blog/cockroachdb-sigmod-2020/?utm_campaign=cooperpress-2020-Q2&utm_source=dbweekly&utm_medium=newsletter&utm_content=dbweekly-primary-sigmod
https://ketanbhatt.com/db-concurrency-defects/
作者把数据结构抽象了，不按照传统的什么树跳表堆什么的，根据用途来分类
其实这种和之前提到的index-handle有点像
这里介绍Bulk Data
作者也没有好的术语来概括他们，大概就是一组数据
更抽象一点
并且每种类型都有很多属性来判断，对于音效
对于bulk data资源，我们假定
可能某些场景顺序也有要求，比如需要渲染的对象，需要从头到尾渲染一遍
作者推荐把需要顺序的捞出来主动排序一下，而不是用个能排序的容器来保存，基数排序O(n) 也能接受（本来数据规模也不大）
至于POD类型，这里假定所有对象都是固定大小的POD，至于有的可能是链表结构这个放到后面数组的数组在讨论
这里还以声音资源来举例
typedef struct {
resource_t * resource; // 资源 引用
uint64_t bytes; // 资源大小
uint64_t format; // 资源的格式判断位
} sound_resource_t;
typedef struct {
sound_resource_t *resource; // 正在播放的资源 引用
uint64_t samples_played; // 播放了的个数
float volume; // 音量
} playing_sound_t;
typedef struct {
playing_sound_t *sound; // 渐入的资源 引用
float fade_from; // 音量
float fade_to; // 音量
double fade_from_ts; // 什么时候开始渐入
double fade_to_ts; // 什么时候结束渐入
} playing_fade_t;
我们对于保存bulk data有这么几个要求
最简单的实现bulk data的方法就是用一个静态数组或者一个vector
#define MAX_PLAYING_SOUNDS 1024
uint32_t num_playing_sounds;
playing_sound_t playing_sounds[MAX_PLAYING_SOUNDS];
// C++ vector
std::vector<playing_sound_t> playing_sounds;
如果你知道对象的个数用静态数组非常见到粗暴，如果不确定，可能浪费或者不够用
使用std::vector要注意一些问题
另外两者都不支持对象引用？？？
如果a[i]被删，几种处理手段
搬迁后面的节点，覆盖空的slot 太浪费时间，除非你要保证顺序（尤其是erase）
直接搬迁最后面的，swap-and-pop
std::swap(a[i], a[a.size() - 1]);
a.pop_back();
直接从后面分配，无脑
留着这个洞，后面直接分配这个 freelist 需要在用slot前检查
作者建议各取所需，除非对遍历有强需求，否则第三种很简单，idindex信息也都在，使用更方便
可能最终维护框架长这个样子
// The objects that we want to store:
typedef struct {...} object_t;
// An item in the free list points to the next one.
typedef struct {
uint32_t next_free;
} freelist_item_t;
// Each item holds either the object data or the free list pointer.
typedef union {
object_t;
freelist_item_t;
} item_t;
typedef struct {
std::vector<item_t> items;
} bulk_data_t;
void delete_item(bulk_data_t *bd, uint32_t i) {
// Add to the freelist, which is stored in slot 0.
bd->items[i].next = bd->items[0].next;
bd->items[0].next = i;
}
uint32_t allocate_slot(bulk_data_t *bd) {
const uint32_t slot = bd->items[0].next;
bd->items[0].next = bd->items[slot].next;
// If the freelist is empty, slot will be 0, because the header
// item will point to itself.
if (slot) return slot;
bd->items.resize(bd->items.size() + 1);
return bd->items.size() - 1;
}
直接用索引信息就可以了
加上版本信息就可以了
typedef struct {
uint32_t id; // index
uint32_t generation; // 引用计数类似物，版本号？
} weak_pointer_t;
typedef struct {
uint32_t generation;
union {
object_t;
freelist_item_t;
};
} item_t;
std::vector本身有内存放大问题，如果满了就会扩容搬迁，这导致了一些问题
解决办法
数组结构体
最终结论 bulk data保存最终方案
There are advantages and drawbacks to everything, but my default recommendation for storing bulk data for a new system would be:
An array of structures, with “holes” and permanent pointers, either allocated as one single large VM reservation (if possible) or as an array of fixed size blocks (of 16 K or whatever is a good fit for your data).
And for the cases where you need really fast number crunching over the data:
A structure of arrays of tightly packed objects, grouped 8 at a time for SIMD processing and allocated as one single large VM reservation, or as an array of fixed-size blocks.
结构体数组 方便管理
数组结构体，可以用simd加速，但是不好管理